Alert Manager Dashboards Not Working Correctly

[cranney117]

My company is using Alert Manager on its Splunk Cloud instance. Since 10/1/2021 we’ve started seeing several issues with Alert Manager dashboards.

On the default Incident Posture dashboard we see errors: “Error in 'SearchParser': Missing a search command before '('. Error at position '2' of search query '| ((index="main" OR index="alerts")) sourcetype="a'.”

We have another dashboard “Incidents Count by Month” which was getting much of its data from source="alert_handler.py". After 10/1 we started seeing less events from source="alert_handler.py" and the events we did get were missing key fields like “label” which were present in events prior to 10/1. Due to the missing label fields the Incidents Count by Month stopped functioning as needed.

I’m wondering if the sudden loss of functionality may be connected to Splunk removing support for Python 2 around the same time, as the source of the data is source="alert_handler.py" it seems a likely culprit.
We are using the (currently)most recent version of Alert Manager 3.0.8, which should resolve any Python issues as far as I know. Would your team be aware of any known solutions for these problems or would you be able to assist in troubleshooting/resolving the sudden change in Alert Manager’s behavior?