Can't receive data, but no error logs are displayed

[spalessandro]

Hi and thanks for your support.
We are trying to use this repo to get Microsoft Teams Call Records into Splunk, since the Microsoft Teams Add-on for Splunk refers to this repo for Splunk Cloud environments.
We deployed the Azure Functions from this repo to our Azure Account.

We linked it to an Azure Application which has CallRecords.Read.All permission, and we made sure to click the Grant admin consent for....

On Splunk side, we created a Splunk HEC input, and filled in the endpoint and token when prompted in the set-up page for the Azure Function. For the HEC endpoint we used the standard form https://http-inputs.<host>.splunkcloud.com:443/services/collector/event.

Finally, we created the subscription as documented. Everything goes through without errors. However, we can't see the data on Splunk. On Splunk, we ran index=_introspection token to troubleshoot the issue: there are no errors, we can just see 0 bytes of data received at any point in time. On all the logs we could find on Azure there are no mentions of errors (except for the subscription creation phase, when it's waiting for the webhook).

We tested the HEC input with a curl POST call from a local machine, and it works: we can see the data on Splunk.

What could be the issue here?
Thanks

[JasonConger]

Were you able to successfully create and list the subscription on the Azure side? https://github.com/splunk/azure-functions-splunk/tree/master/graph#3-create-a-microsoft-graph-subscription

[kclifford20]

@spalessandro Did you find a fix for this? We've got the same issue.
I can see in the logs on the azure function side that the queue is getting call records and being processed, however we can't see anything on the Splunk search side.

[spalessandro]

@JasonConger We succesfully created the subscription. We get as output:
[create-subscription] successfully created subscription: { "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#subscriptions/$entity", "id": "xxxxxx", "resource": "/communications/callRecords", "applicationId": "xxxxxxxx", "changeType": "created,updated", "clientState": null, "notificationUrl": "xxxxxxxxx", "notificationQueryOptions": null, "lifecycleNotificationUrl": null, "expirationDateTime": "2022-06-05T09:41:59.483Z", "creatorId": "xxxxxxxxxx", "includeResourceData": null, "latestSupportedTlsVersion": "v1_2", "encryptionCertificate": null, "encryptionCertificateId": null, "notificationUrlAppId": null }

When we list the subscriptions we get a similar output, with a list of several subscriptions among which appears one with the same id as the one created.

@kclifford20 No, we still haven't found a fix. Still no data on Splunk. Could you please share where did you find the logs on azure about the queue getting call records? Thanks

[kclifford20]

@kclifford20 No, we still haven't found a fix. Still no data on Splunk. Could you please share where did you find the logs on azure about the queue getting call records? Thanks

@spalessandro A couple of places. If you go to your resource group, you'll see a storage account with the name ending in functions. Open that up, then queues, notification-queue. Make sure that shows no results.
Then go back to your function app, functions, process-notification-queue, Monitor. Check that there are invokations listed in there.

[kclifford20]

Were you able to successfully create and list the subscription on the Azure side? https://github.com/splunk/azure-functions-splunk/tree/master/graph#3-create-a-microsoft-graph-subscription

I have a feeling it's to do with Splunk having invalid certificates on the HEC due to a double . used in the DNS name causing the initial wildcard to not be valid.
For instance, the HEC URL is https://http-inputs.__customertenant__.splunkcloud.com:443/services/collector/event
The SSL certificate is *.splunkcloud.com, therefore it'll only work for customertenant.splunkcloud.com

[kclifford20]

@spalessandro With some help from our Splunk reps, we've found a fix. The problem is that the default HEC inputs URL has a SSL certificate issue. This blocks the scripts from working. The fix is to replace the . between http-inputs and customertenant with a -

For example: https://http-inputs-__customertenant__.splunkcloud.com/services/collector/event

This is working for us

[kclifford20]

@JasonConger Could you please update the docs with the example HEC URL?

[spalessandro]

@kclifford20 Thanks for the insight, I'll try that and report if it works.

[spalessandro]

@kclifford20 Thank you with all my heart, I've been spending a lot of time on this error and you just solved it :D
@JasonConger I agree with @kclifford20 the docs should be updated.
Thanks!

[vasudevkadival]

Hi All
We have a similar issue with our deployment, we have done all the configurations from Splunk and Azure end but we are not receiving any data and there is no error message from Splunk.
When we check the "Process Notification Queue" on Azure Functions, it shows "No inputs defined" and "No outputs defined"
We tried updating the Splunk HEC Endpoint URL as suggested in above thread, but no luck
Could someone please share a solution to this ?