Skip to content

Latest commit

 

History

History
211 lines (132 loc) · 11.2 KB

troubleshoot-telemetry-endpoint.md

File metadata and controls

211 lines (132 loc) · 11.2 KB
title description author ms.author ms.reviewer ms.date ms.topic
Troubleshoot connectivity to data processing service and telemetry endpoints
Describes how to troubleshoot connectivity to the data processing service (DPS) and telemetry endpoints.
twright-msft
twright
mikeray
03/25/2025
troubleshooting

Troubleshoot connectivity to the data processing service and telemetry endpoints

[!INCLUDE sqlserver]

In addition to the usual endpoints, the Azure Arc extension for SQL Server connects to two other endpoints:

  • Data processing service (DPS) endpoint

    The collected inventory information about SQL Server instances, databases, availability groups, and usage data for billing purposes is sent to this endpoint.

  • Telemetry endpoint

    The Azure Connected Machine agent logs, the Azure extension for SQL Server logs, and the Dynamic Management Views (DMV) data is sent to this endpoint.

Communication to these endpoints uses HTTPS with SSL/TLS and port TCP/443 for encrypted secure connections. The agent initiates communication to send the data to Azure. Azure never initiates communication. Connectivity to these endpoints is therefore only one way.

When communication to these endpoints is blocked, the service has the following symptoms:

Azure extension current state

You can view the current state of the Azure extension for SQL Server in the portal. The status is refreshed every 15 minutes.

:::row::: :::column:::

Healthy state:

:::image type="content" source="media/troubleshoot-telemetry-endpoint/healthy-state.png" alt-text="Screenshot of portal for Azure extension for SQL Server in a healthy state.":::
    
:::column-end:::
:::column:::

Unhealthy state:

:::image type="content" source="media/troubleshoot-telemetry-endpoint/unhealthy-state.png" alt-text="Screenshot of portal for Azure extension for SQL Server in an unhealthy state.":::
    
:::column-end:::

:::row-end:::

Check if you have a problem connecting to the DPS or telemetry endpoints

There are two ways to check if you have connectivity problems to the DPS or telemetry endpoints.

Check the Azure Extension for SQL Server status in the Azure portal

If it's connected to Azure in general, the Azure Extension for SQL Server reports its status in the Azure portal.

  • Navigate to the Machines - Azure Arc view in the Azure portal and locate the machine by name and select it.
  • Select Extensions.
  • Select WindowsAgent.SqlServer or LinuxAgent.SqlServer to bring up the details.
  • Look at the Status message and the uploadStatus value. If it's anything other than OK, there's a problem with connecting to the DPS. If it's 0, it's likely that there's a firewall blocking the communication to the DPS endpoint. There could be more details in the status message or the uploadStatus error code that can provide insights into the connectivity problem.

Check the Azure Extension for SQL Server logs

The extension log file is at:

C:\ProgramData\GuestConfig\extension_logs\Microsoft.AzureData.WindowsAgent.SqlServer\

The log file name depends on the version Azure Extension for SQL Server. For the latest version of Azure Extension for SQL Server, the log file is:

unifiedagent.log

For extension version 1.1.24724.69 and earlier, the log file is:

ExtensionLog_0.log

Check for log entries that indicate a problem connecting to the DPS or telemetry endpoints.

Probe web server endpoints

You can use various tools to probe the web server endpoints for DPS and telemetry. For example, Invoke-WebRequest or curl.

The following example uses Invoke-Webrequest:

Invoke-WebRequest telemetry.<region>.arcdataservices.com

A possible response status code is:

Invoke-WebRequest: Response status code does not indicate success: 401 (Unauthorized).

401 is expected because there is no unauthenticated route on the telemetry endpoint.

For DPS:

Invoke-WebRequest dataprocessingservice.<region>.arcdataservices.com

A possible response status code is:

StatusCode        : 200

StatusDescription : OK

This one should return a 200 as there is an unauthenticated route.

Probe connectivity to all regions

You can probe connectivity to all regions with the test-connectivity.ps1 PowerShell script.

:::code language="powershell" source="~/../sql-server-samples/samples/features/azure-arc/troubleshooting/test-connectivity.ps1":::

Check TLS version compatibility

The data processing service endpoint supports the following TLS versions: TLS 1.2 and 1.3. Windows Server 2012 and older versions aren't supported.

For telemetry endpoints, Windows Server 2012 R2 and older are not supported.

If an unsupported TLS version is being used, you may see an error in the log

<date time>|ERROR|SqlServerExtension.Service|Request failed with exception 'System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.

---> System.Security.Authentication.AuthenticationException: Authentication failed because the remote party sent a TLS alert: 'HandshakeFailure'.

---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.

Endpoint reference

To connect to Azure, the endpoints use *.arcdataservices.com.

Additional background

Beginning with March, 12 2024, the Azure Extension for SQL Server uses the following endpoints:

  • DPS: dataprocessingservice.<region>.arcdataservices.com
  • Telemetry telemetry.<region>.arcdataservices.com

Replace <region> with the short name of the Azure region where the Arc machine resource is located. The short name is derived from the Azure region name without spaces and all lower case.

For example, if your Arc machine resource is located in East US 2 the short name of the region is eastus2 and the telemetry endpoint is:

telemetry.eastus2.arcdataservices.com

If your extension is older than March 11, 2024, it may use older endpoints. Update your extension to use the current endpoint.

Note

The endpoint values before *.arcdataservices.com are subject to change.

Use an HTTPS proxy server for outbound connectivity

If your network requires using an HTTPS proxy server for outbound connectivity, you can read more about configuring that at Update or remove proxy settings.

Query Azure Resource Graph for telemetry upload stats

Use Azure Resource Graph to query the upload status for your environment.

resources
    | where type =~ 'microsoft.hybridcompute/machines/extensions'
    | where properties.type in ('WindowsAgent.SqlServer','LinuxAgent.SqlServer')
    | parse id with * '/providers/Microsoft.HybridCompute/machines/' machineName '/extensions/' *
    | parse properties with * 'uploadStatus : ' uploadStatus ';' *
    | project uploadStatus, subscriptionId, resourceGroup, machineName
    | where uploadStatus !in ('OK') //comment this out to see all upload stats
    | order by uploadStatus desc

Find SQL extensions that have not connected to DPS in a long time

Query Azure Resource Graph to find extensions that have not connected to DPS recently.

:::code language="powershell" source="~/../sql-server-samples/samples/features/azure-arc/troubleshooting/hybrid-compute-extension-last-connect.kql":::

Error codes

The following table shows some of the common DPS upload status values and what you can do to troubleshoot further.

DPS upload status value HTTP error code Troubleshooting suggestions
0 Likely cause: a firewall is blocking the transmission of the data to the DPS. Open the firewall to the DNS endpoint for the DPS (TCP, port: 443).
OK 200 The connection is working as expected.
Bad request 400 Possible cause: The resource name (SQL Server instance or database name) doesn't conform to Azure resource naming conventions. For example, if the database name is a reserved word.
Unauthorized 401 Likely cause: the extension is configured to send data through an HTTP proxy that requires authentication. Using an HTTP proxy that requires authentication is not currently supported. Use an unauthenticated HTTP proxy or no proxy.
Forbidden 403 If the Azure Connected Machine agent is otherwise working as expected and this error doesn't resolve itself after a reboot, create a support case with Microsoft Support through the Azure portal.
NotFound 404 The endpoint that the extension is trying to connect to doesn't exist.

To check which endpoint it is trying to connect to, search the logs for dataprocessingservice. This condition can happen if the Azure Connected Machine agent was deployed and connected to an Azure region in which the Microsoft.AzureArcData resource provider is not yet available. Redeploy the Azure Connected Machine agent in a region that the Microsoft.AzureArcData resource provider for aSQL Server enabled by Azure Arc is available. See also Region availability.

It is possible that the DNS resolver cache is not refreshed for your machine. To refresh:
- On Windows run: ipconfig /flushdns
- On Linux (if systemd is being used) run: sudo resolvectl flush-caches
Conflict 409 Likely cause: temporary error happening inside of the DPS. If this does not resolve itself, create a support case with Microsoft Support through the Azure portal.
InternalServerError 500 This is an error that is happening inside of the DPS. Create a support case with Microsoft Support through the Azure portal.

Related content