ServiceClient is not connected to Dataverse. Please recreate this ServiceClient

[gonzaloenterrios]

Hi,

We are having some random issues with the ServiceClient connections when using it from our service bus queue triggered Azure Functions. These are hosted on an App Service Plan Premium v3 P1V3, with Always On enabled, no auto scaling and one 1 instance.
We are getting the error "ServiceClient is not connected to Dataverse. Please recreate this ServiceClient". Looking at the code, I see that happens when the ValidateConnectionLive method is called prior to any call to Dataverse. So my guess is that the connection works fine for a while, but then at some point the IsReady is set to false and because the _connectionOptions is null we get that error.

internal void ValidateConnectionLive()
{
if (isDisposed)
{
// This client has been disposed and no property of it can be trusted. Thrown Exception should bubble to caller.
throw new ObjectDisposedException("This instance of the ServiceClient has been disposed and cannot be used. You must create a new ServiceClient instance.");
}

        if (!this.IsReady)
        {
            if (this._connectionOptions != null)
            {
                var failureExecpt = new DataverseConnectionException("ServiceClient is Staged for Connection but not Connected. You must call .Connect() on this client before using it.");
                _logEntry?.Log(failureExecpt);
                throw failureExecpt;
            }
            else
            {
                var failureExecpt = new DataverseConnectionException("ServiceClient is not connected to Dataverse. Please recreate this ServiceClient.");
                _logEntry?.Log(failureExecpt);
                throw failureExecpt;
            }
        }
    }

At the beginning of our function code, we call the constructor ServiceClient(X509Certificate2 certificate, StoreName certificateStoreName, string certificateThumbPrint, Uri instanceUrl, bool useUniqueInstance, OrganizationDetail orgDetail, string clientId, Uri redirectUri, ILogger logger = null) and we used the same instance through the entire execution. The useUniqueInstance parameter is set to false, so it means we are reusing the cached connection if any.

My question is, what can cause the connection to go down? Should we implement a retry mechanism to catch DataverseConnectionException exceptions and authenticate again? Or is there something we can change on the code to avoid those issues?

Please let me know if you need further information to assess the issue.

Thanks

[MattB-msft]

Please make sure you're running the current version of the DVSC Client.

The only thing that will invalidate the DVSC client in that way is if you dispose the client.
Are you using the client inside a "using statement" anywhere?

In general, you should not need to set the useUniqueInstance and should dispose your client after each rotation.

[gonzaloenterrios]

Hi,

We are not using a "using statement", but just instantiating the ServiceClient and the beginning of our RunAsync method. So I don't understand then how it can get disposed at the middle of the code execution.
Important to remark that this happens 1 to 10 times per day and only during the daily synchronisation process. We publish around 30k messages on the queue that are processed by the Azure Function and then we get a few times this error with the ServiceClient. So I'm assuming it is somewhat related with the load or performance.

Please let me know if you need further details.

Regards,

[MattB-msft]

Sorry for the delay here... Is there a way you can share your connection code and or sample of how your using this ?
Are you using DI?

thanks

[FarmerenHE]

Important to remark that this happens 1 to 10 times per day and only during the daily synchronisation process. We publish around 30k messages on the queue that are processed by the Azure Function and then we get a few times this error with the ServiceClient. So I'm assuming it is somewhat related with the load or performance.

A few years back I had a requirement to make an API that could hand 1.000 request / second to dataverse, we found back then that some times the connection would drop without any obvious error and the most likely culprint was an issue with authentication to Azure AD that dropped for no apparant reason. - Also just 1-10 times every now and then. (That was on the .Net framework version of the SDK).

We "solved" it by making a wrapper IOrganizationServiceAsync2 and then have every call in a try/catch that retries the call one time before throwing the error upwards, that solved the issue for us.