Bound the node registry reads used by the cluster health checks

[patrickmann]

Impact

Two checks in the System Overview health panel, Certificates and OpenSearch Version (both under Search Cluster), can be blocked on MongoDB reads that the health plugin cannot put a time limit on.

When one of those reads is slow or never returns, the check reports unknown instead of a definite state. An unknown check is dropped from the panel's rollup unless everything beside it is also unknown, so the section above it can stay green while a real problem goes unreported.

Why the plugin cannot fix this itself

Each check gets a fixed time budget, and the framework can only ignore a late answer, not stop the work behind it. So every check has to impose its own limit on the call it makes. Most MongoDB-backed checks do that with the driver's withTimeout, which bounds the whole operation including retries. These two cannot, because the collection is a private field rather than something the caller can reach.

What to do

Give each service a bounded read that a caller on a deadline can use. Either a timeout-carrying overload alongside the existing method, or expose a bounded read the callers can drive. Keep it additive so existing callers are untouched.

Dependencies

None. This touches AbstractNodeService and DataNodeMetadataServiceImpl only, and conflicts with nothing currently in flight. It is deliberately split out of #26886, whose ClusterAdapter half is blocked on #26885 and touches the three search client classes instead.

Consumers

Tracked on the Enterprise side in Graylog2/graylog-plugin-enterprise#15110, under "Bounded node registry reads".

[tellistone]

What is the impact of this?

[patrickmann]

@tellistone Node registry reads don't meet the Health Panel spec requirement of a fixed timeout; and cannot use the framework that other reporters use (for technical reasons). This issue was created to bring the affected reporters into compliance.

Just curious, why wasn't that clear enough from the original description? If you ignore the technical gobbledigook, it says just that:

The Enterprise health framework gives each cluster reporter a 5s wall-clock budget ... Most MongoDB-backed reporters comply using the driver's withTimeout, which covers retries. These two cannot

I appreciate that triage doesn't want to go chasing links to related issues to get a proper understanding.

[tellistone]

Just curious, why wasn't that clear enough from the original description?

Personally I'm not familiar with the code, so I don't follow the meaning of eg. "SPI therefore requires each reporter to bound its own backend call." or the significance of the variables detailed. For ease of triage, ideally we'd have a description of the problem that works backwards from how it manifests as a problem for the end user. eg:

Two of the reporters on the health panel can be blocked on MongoDB reads that the plugin cannot bound. This impacts the Data Node Certificate state, and Data Node Version checks.

Each has a 5s wall-clock budget and can only abandon a result, never kill the worker. This results in an "unknown" status for the affected checks when timeouts happen, which can hide an actual problem (esp if the problem is the cause of the timeouts).

[patrickmann]

Makes sense. This particular issue was more of a technical task that needed completing and targeted at engineers. But I get that you need to be able to at least understand how to prioritize it against other issues.