new tracing: documentation, changelogs, linting, update test data

Author: mgmeier

bench/tx-generator/CHANGELOG.md

@@ -1,10 +1,17 @@
 # ChangeLog
 
+## 2.15 -- Mar 2025
+
+* A new cabal flag `withplutuslib` is added, enabling import and re-compilation of Plutus scripts from `plutus-scripts-bench` - default: false; use for dev/test of new benchmarks only.
+* Port `ProtocolParameters` type and typeclass instances from `cardano-api` into new module `Cardano.Api.Internal`, removing dependency on the deprecated API type.
+* Bump for Node 10.3
+
 ## 2.14.2 -- Oct 2024
 
 * Bump for Node 10
 
 ## 2.14.1 -- June 2024
+
 * A new NixSvcOptions field is introduced: `_nix_keepalive`
   and it's propagated down to the `kaClient` that does keepalives.
   This makes keepalive timeouts configurable.

bench/tx-generator/src/Cardano/Benchmarking/Tracer.hs

@@ -67,12 +67,8 @@ generatorTracer ::
   -> Maybe (Trace IO FormattedMessage)
   -> IO (Trace IO a)
 generatorTracer tracerName mbTrStdout mbTrForward = do
-  forwardTrace <- case mbTrForward of
-                        Nothing -> mempty
-                        Just trForward -> forwardFormatter trForward
-  stdoutTrace  <- case mbTrStdout of
-                        Nothing -> mempty
-                        Just trForward -> machineFormatter trForward
+  forwardTrace <- maybe mempty forwardFormatter mbTrForward
+  stdoutTrace  <- maybe mempty machineFormatter mbTrStdout
   let tr = forwardTrace <> stdoutTrace
   tr'  <- withDetailsFromConfig tr
   pure $ withInnerNames $ appendPrefixName tracerName tr'

cardano-node/test/Test/Cardano/Tracing/NewTracing/data/badConfig.yaml

@@ -72,12 +72,6 @@ TraceOptions:
   TxSubmission.Remote:
     detail:   DMinimal
 
-# Options for the trace forwarder
-TraceOptionForwarder:
-  address:
-    filePath: /tmp/forwarder.sock
-  mode: Initiator
-
 TraceOptionPeerFrequency: 2000
 
 TraceOptionResourceFrequency: 5000

cardano-node/test/Test/Cardano/Tracing/NewTracing/data/goodConfig.yaml

@@ -69,12 +69,6 @@ TraceOptions:
   TxSubmission.Remote:
     detail:   DMinimal
 
-# Options for the trace forwarder
-TraceOptionForwarder:
-    address:
-      filePath: /tmp/forwarder.sock
-    mode: Initiator
-
 TraceOptionPeerFrequency: 2000
 
 TraceOptionResourceFrequency: 5000

cardano-tracer/CHANGELOG.md

@@ -1,5 +1,14 @@
 # ChangeLog
 
+## 0.3.2 (March 2025)
+
+* When requesting forwarded metrics, ask for delta to previous request only. New config option `ekgRequestFull` defaults to `false`; set to `true` to revert this behavior.
+* Add up-to-date symlink for each log rotation, pointing to the most recent file (`node.json` or `node.log`, depending on the log format).
+* Fix: CLI option `--min-log-severity` is now observed correctly.
+* Add `metricsNoSuffix` to configuration, which when set removes suffixes like `_int` from metrics names, making them more similar to those in the old system (optional boolean; default: `false`).
+* Remove `metricsComp` metric name remapping from configuration.
+* Updated to use `ekg-forward-0.9`.
+
 ## 0.3.1 (January 22, 2025)
 
 * Fix race condition when querying `NodeInfo` data point, occasionally resulting in fallback `NodeName`s instead of proper ones.

cardano-tracer/docs/cardano-tracer.md

@@ -296,7 +296,7 @@ In this example, `subdirForNode0` is a subdirectory containing log files with it
 
 The field `logMode` specifies logging mode. There are two possible modes: `FileMode` and `JournalMode`. `FileMode` is for storing logs to the files, `JournalMode` is for storing them in `systemd`'s journal. Please note that if you choose `JournalMode`, the field `logRoot` will be ignored.
 
-The field `logFormat` specifies the format of logs. There are two possible modes: `ForMachine` and `ForHuman`. `ForMachine` is for JSON format, `ForHuman` is for human-friendly text format.
+The field `logFormat` specifies the format of logs. There are two possible modes: `ForMachine` and `ForHuman`. `ForMachine` is for JSON format, `ForHuman` is for human-friendly text format. The latter also implies the file endings change to `.log`, and the symbolic link is named `node.log`.  
 
 Please note that `logging` field accepts the list, so you can specify more than one logging section. For example, for both log formats:
 

trace-dispatcher/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Revision history for trace-dispatcher
 
+## 2.9 -- Mar 2025
+
+* New `PrometheusSimple` backend which runs a simple TCP server for direct exposition of metrics, without forwarding.
+* New `maxReconnectDelay` config option in `TraceOptionForwarder`: Specifies maximum delay (seconds) between (re-)connection attempts of a forwarder (default: 60s).
+* Optimize data sharing in formatters.
+* Remove unused optional namespace prefix argument from formatters.
+* Updated to use `ekg-forward-0.9`.
+* Remove `ekg-wai` from dependencies.
+
 ## 2.8.1 -- Feb 2025
 
 * Updated to `ouroboros-network-framework-0.16`

trace-dispatcher/doc/config.nix

@@ -15,4 +15,3 @@ To add in node-services.nix
                  {ns = ""; backends = ["Stdout HumanFormatColoured"; "Forwarder"; "EKGBackend"];}
                  {ns = "ChainDB"; backends = ["Forwarder"];}
                ];
-               TraceOptionForwarder = {filePath: "/tmp/forwarder.sock";};

trace-dispatcher/doc/config.yaml

@@ -61,12 +61,6 @@ TraceOptions:
   BlockFetchClient.CompletedBlockFetch:
     maxFrequency: 2.0
 
-# Options for the trace forwarder
-TraceOptionForwarder:
-    address:
-      filePath: /tmp/forwarder.sock
-    mode: Initiator
-
 TraceOptionPeerFrequency: 2000
 
 TraceOptionResourceFrequency: 5000

trace-dispatcher/doc/trace-dispatcher.md

@@ -305,8 +305,19 @@ The configurability of dispatchers provided by this library relies on:
 The usual form to provide a configuration is via a configuration file, wich can be in JSON or YAML format. The options that
 can be given based on a namespace are: `severity`, `detail`, `backends` and `limiter`.
 
-Backends can be a combination of `Forwarder`, `EKGBackend`, and
-one of `Stdout MachineFormat`, `tdout HumanFormatColoured` and `Stdout HumanFormatUncoloured`.
+Backends can be a combination of `Forwarder`, `EKGBackend`, `PrometheusSimple [suffix|nosuffix] [bindhost] <port>` and
+one of `Stdout MachineFormat`, `Stdout HumanFormatColoured` and `Stdout HumanFormatUncoloured`.  
+
+The connection for the `Forwarder` backend is provided on the application command line. It is a socket path over which applications like `cardano-node` connect with `cardano-tracer`. `--tracer-socket-path-connect /path/to/forward.sock` sets
+the backends's role to `Initiator`, whereas `--tracer-socket-path-accept /path/to/forward.sock` sets it to `Responder`. Except for debugging purposes, the former should be chosen: the application takes the `Initiator` role, and `cardano-tracer` is
+in the `Responder` role, which means setting its network `tag` to `AcceptAt` in its config (see there).  
+
+The `PrometheusSimple` backend provides Prometheus metrics _directly from the process_, without forwarding. It always applies to all tracers globally, and should only be configured once.
+Providing an available port number in the connection string is mandatory; this will bind to localhost only by default. By specifying a bind host, the metrics can be queried remotely, e.g. over IPv4 by
+binding to `0.0.0.0`, or IPv6 by binding to `::`. Metrics will be available under the URL `/metrics`. 
+The `nosuffix` modifier removes suffixes like `_int` from metrics names, making them more similar to those in the old system; `suffix` is the implicit default and can be omitted.
+
+*CAUTION*: Generally allowing remote queries of Prometheus metrics is risky and should only be done in an environment you control.
 
 ```yaml
 # Use new tracing
@@ -320,6 +331,7 @@ TraceOptions:
       - Stdout MachineFormat
       - EKGBackend
       - Forwarder
+      - 'PrometheusSimple :: 1234' # Prometheus metrics available over IPv6 (and localhost) on port 1234
 
   ChainDB: # Show as well messages with severity Info for all ChainDB traces.
     severity: Info
@@ -329,16 +341,13 @@ TraceOptions:
     maxFrequency: 2.0
 
 TraceOptionForwarder: # Configure the forwarder
-    address:
-      filePath: /tmp/forwarder.sock
-    mode: Initiator
     maxReconnectDelay: 20
 
 # Frequency of Peer messages set to two seconds
 TraceOptionPeerFrequency: 2000
 
 # Any metrics emittted will get this prefix
-TraceOptionMetricsPrefix: "cardano.node"
+TraceOptionMetricsPrefix: "cardano.node.metrics."
 ```
 
 The same in JSON looks like this:
@@ -353,7 +362,8 @@ The same in JSON looks like this:
       "backends": [
         "Stdout MachineFormat",
         "EKGBackend",
-        "Forwarder"
+        "Forwarder",
+        "PrometheusSimple :: 1234"
       ]
     },
     "ChainDB": {
@@ -365,14 +375,10 @@ The same in JSON looks like this:
     }
   },
   "TraceOptionForwarder": {
-    "address": {
-      "filePath": "/tmp/forwarder.sock"
-    },
-    "mode": "Initiator",
-    "maxReconnectDelay": 30
+    "maxReconnectDelay": 20
   },
   "TraceOptionPeerFrequency": 2000,
-  "TraceOptionMetricsPrefix": "cardano.node"
+  "TraceOptionMetricsPrefix": "cardano.node.metrics."
 }
 ```
 
@@ -708,7 +714,7 @@ As mentioned earlier, trace backends serve as the final destinations for all tra
 
     ```haskell
     ekgTracer :: MonadIO m
-      => Either Metrics.Store Server
+      => Metrics.Store
       -> m (Trace m FormattedMessage)
     ```
 

trace-forward/CHANGELOG.md

@@ -1,5 +1,9 @@
 # ChangeLog
 
+## 2.2.11 - Mar 2025
+
+* Updated to `ouroboros-network-framework-0.17`
+
 ## 2.2.10 - Feb 2025
 
 * Updated to `ouroboros-network-framework-0.16`