We should identify additional metrics that can be used to reason about server "capacity"

[jhiemstrawisc]

@patrickbrophy and @brianaydemir, feel free to chime in if you think I'm misrepresenting any of our conversation.

Right now, the Director's only real notion for "is server X at/nearing its capacity to serve clients" comes from checking the server's "status" field, which can set statuses like "degraded" or "critical".

At the moment, a server's "degraded" status is only triggered when the current value of the xrootd_server_active_io metric reaches some threshold relative to the limit configured via {Cache,Origin}.Concurrency. But we've started to identify other ways in which servers can reach capacity. For example, a Cache/Origin is at/nearing capacity when:

• XRootD is unable to spawn additional threads because it has hit the limit configured with Xrootd.MaxThreads.
• the server's total available bandwidth is in use such that additional clients result in reduced bandwidth per client
• the server is running out of memory (not sure we've observed this, but it's a useful example)

A key observation is that there are multiple "axes" of capacity, and that they can be observed independently even if some of them are related. Moving forward, we should try to identify as many of these as possible to start using them as triggers for degraded or critical statuses.

I'm spitballing here, but I think any metric is a good candidate when it meets these criteria:

• it directly correlates to the usage of a limited resource that, when exhausted, causes "poor performance" (@brianaydemir, I think explaining what "poor performance" encompasses means describing the failure mode, such as when you pointed out that hitting a thread limit will cause instant failures whereas bandwidth issues get gradually worse -- we can probably use this to decide whether something constitutes a critical or a degraded status.)
• we can determine a rough upper bound on the amount of that resource that's available to the server, either programmatically or via admin-supplied configuration (and we can provide the admin guidance for how to set a limit if it's their responsibility).
• we can determine when the server reaches some usage threshold for the resource relative to the total available

Is there anything I'm missing here?

Finally, @patrickbrophy pointed out we should revisit the corpus of XRootD monitoring statistics available to us that we don't currently plumb into Pelican to see whether any of them might be useful for this exercise.

[brianaydemir]

the server's total available bandwidth is in use such that additional clients result in reduced bandwidth per client

An official Pelican client is perfectly capable of conveying its Client.MinimumDownloadSpeed, so a cache or origin shouldn't need to guess what quality-of-service it needs to deliver. (And I'm sure there are other network- and connection-related parameters that would be equally-useful to convey.)

I think explaining what "poor performance" encompasses means describing the failure mode, such as when you pointed out that hitting a thread limit will cause instant failures whereas bandwidth issues get gradually worse

Correct. For example, we want a cache to be using all of the network bandwidth that it believes it's been given, but we also want it to allocate that bandwidth in such a way that it meets clients' minimum requirements. If it doesn't belive that it can meet a specific client's requirement, it should fail the attempt early and loudly.

[patrickbrophy]

During some sprucing up of the metrics docs, I was reading through the summary stats that we get from XRootD, in particular these stats look promising. Currently we are able to observe the bytes sent in/out and but things like read timeouts and active connections seem like they could be really helpful.

With active connections we could expose this as a prometheus gauge. We could then allow for admins to configure how many total active clients can connect and the rate of connection.

With the read timeouts, we can deprioritize the services that have the highest rate of timeouts.