Feature preview: built-in OpenMetrics exporter

[abower-amd]

Recent commits to sfptpd add built-in OpenMetrics stats exporting. This can be used to populate a time series database like Prometheus and then be visualised on a dashboard, like this:

The stats currently exported are the 'real time stats' that can already be accessed via the json_stats option but can now be fed directly to a time series database without being tailed and processed by an intermediary exporter. For now, sfptpd listens on a Unix domain socket (/var/run/sfptpd/metrics.sock by default) so a proxy is needed; nginx can do that, or you can use socat like this (See man sfptpd):

socat TCP-LISTEN:9979,fork UNIX-CONNECT:/var/run/sfptpd/metrics.sock

For a quick preview of the stats, use:

curl --unix-socket /var/run/sfptpd/metrics.sock http://metrics/metrics

Any feedback on this feature would be much appreciated, please! What would you like to see?

There is an example Grafana dashboard.

Some configuration options are available (see sfptpd --help), e.g. to begin each metric family name with sfptpd_, use openmetrics_prefix sfptpd_.

The real time stats samples are timestamped by sfptpd so that users can access detailed history, but additionally there are some equivalent metric families named *_snapshot* which are the last RT sample repeated but without an sfptpd timestamp so that they get timestamped by the ingestor instead.

[sasharozenson]

This is excellent work and looks really solid.

Thank you for the effort you’ve put into this @abower-amd ! This is a significant step forward. It will take me some time before I can potentially migrate my setup from @natesales' sfptpd-exporter, but this looks very promising.

Initial Observations (First 10 Minutes of Testing)

Metric Names

• The metric names don’t currently match the mentioned product, but this is something I can adjust on my end.

Grafana Dashboard

• The provided Grafana dashboard doesn’t render the same on my setup. I’d suggest redesigning it to include:
  • Indicators for NTP or Chrony fallback and the current status (e.g., ptp-slave, listening, ntp-slave, etc.).
  • Alarms, which could be taken or inspired from the sfptpd-exporter project.

System Clock Servo

• There seems to be no way to pull information about which servo is driving the system clock.
  This is crucial for my use case, as I need to monitor the system clock independently. There are situations where:
  • ptp-slave is perfectly in sync, but the system clock drifts significantly.
  • Stepping the system clock is not always feasible since some applications are too sensitive to time steps.
    I rely on step-at-startup option, but currently considering if we can change to slew-and-step.

[abower-amd]

Hi @sasharozenson,

Thanks for trying this out and for your very helpful feedback!

Metric Names

* The metric names don’t currently match the mentioned product, but this is something I can adjust on my end.

An initial goal was to start off as a drop-in replacement for sfptpd-exporter but to comply with the OpenMetrics spec, including appended type names, we made some changes. We also dropped the default sfptpd_ prefix for shorter names but you can add that back in with the openmetrics_prefix configuration option. We renamed the instance label to sync so it didn't clash with Prometheus's automatic labeling of sources. Is there any particular aspect of the metrics naming that bothers you most?

Grafana Dashboard

* The provided Grafana dashboard doesn’t render the same on my setup. I’d suggest redesigning it to include:
  
  * Indicators for NTP or Chrony fallback and the current status (e.g., `ptp-slave`, `listening`, `ntp-slave`, etc.).
  * Alarms, which could be taken or inspired from the `sfptpd-exporter` project.

We are hoping end users will be able to do a better job than our example dashboard! :-)

For the first GA release of this feature we will be focused on doing a solid job presenting what can currently be obtained from the JSON RT Stats as this will enable users to swap out fragile processing pipelines. So that doesn't include instance state information. It does include alarms though. We present the alarms count and an alarms string (as a label). There is also an optional alarms 'StateSet', which is how the OpenMetrics standard suggests we do this sort of thing, but I found it was not easy to present in Grafana so you have to configure it with openmetrics_options.

We can then extend to presenting other information that comes from different internal sources, e.g.

• the 'long term stats' collections
• various counters
• sync source selection information.

Any suggestions on the most useful extra information would be appreciated. You have suggested instance state.

System Clock Servo

* There seems to be no way to pull information about which servo is driving the system clock.
  This is crucial for my use case, as I need to monitor the system clock independently. There are situations where:

You are right, this is missing. We hadn't decided the best way to present this. We will add the information in when we have decided the best way.

(Edited:)
We'll add an Info-style metric with information about the servo. I wasn't necessarily planning to include the master or slave times - they are not that useful I think and perhaps shouldn't even feature in the JSON - except perhaps as an info metric needs a numerical value so we could use that anyway instead of 1.

offset_seconds{sync="servo0"} -0.000000011250 1738066117.397592781
servo_info{sync="servo0,clock="system",master="phc0(eth0/eth1)",primary-intf="eth0"} 1

[abower-amd]

New metrics added:
m_time_seconds
s_time_seconds
servo_info
Thanks @sasharozenson - hope you can use these effectively!