observability.adoc

Observability Support

Micrometer defines an Observation concept that enables both Metrics and Traces in applications. Metrics support offers a way to create timers, gauges or counters for collecting statistics about the runtime behavior of your application. Metrics can help you to track error rates, usage patterns, performance and more. Traces provide a holistic view of an entire system, crossing application boundaries; you can zoom in on particular user requests and follow their entire completion across applications.

Spring Framework instruments various parts of its own codebase to publish observations if an ObservationRegistry is configured. You can learn more about {docs-spring-boot}/html/actuator.html#actuator.metrics[configuring the observability infrastructure in Spring Boot].

Micrometer Observation concepts

If you are not familiar with Micrometer Observation, here’s a quick summary of the new concepts you should know about.

• Observation is the actual recording of something happening in your application. This is processed by ObservationHandler implementations to produce metrics or traces.

• Each observation has a corresponding ObservationContext implementation; this type holds all the relevant information for extracting metadata for it. In the case of an HTTP server observation, the context implementation could hold the HTTP request, the HTTP response, any Exception thrown during processing…​

• Each Observation holds KeyValues metadata. In the case of a server HTTP observation, this could be the HTTP request method, the HTTP response status…​ This metadata is contributed by ObservationConvention implementations which should declare the type of ObservationContext they support.

• KeyValues are said to be "low cardinality" if there is a low, bounded number of possible values for the KeyValue tuple (HTTP method is a good example). Low cardinality values are contributed to metrics only. High cardinality values are on the other hand unbounded (for example, HTTP request URIs) and are only contributed to Traces.

• An ObservationDocumentation documents all observations in a particular domain, listing the expected key names and their meaning.

Configuring Observations

Global configuration options are available at the ObservationRegistry#observationConfig() level. Each instrumented component will provide two extension points:

• setting the ObservationRegistry; if not set, observations will not be recorded and will be no-ops

• providing a custom ObservationConvention to change the default observation name and extracted KeyValues

Using custom Observation conventions

Let’s take the example of the Spring MVC "http.server.requests" metrics instrumentation with the ServerHttpObservationFilter. This observation is using a ServerRequestObservationConvention with a ServerRequestObservationContext; custom conventions can be configured on the Servlet filter. If you would like to customize the metadata produced with the observation, you can extend the DefaultServerRequestObservationConvention for your requirements:

code:ExtendedServerRequestObservationConvention

If you want full control, you can then implement the entire convention contract for the observation you’re interested in:

code:CustomServerRequestObservationConvention

You can also achieve similar goals using a custom ObservationFilter - adding or removing key values for an observation. Filters do not replace the default convention and are used as a post-processing component.

code:ServerRequestObservationFilter

You can configure ObservationFilter instances on the ObservationRegistry.

HTTP Server instrumentation

HTTP server exchanges observations are created with the name "http.server.requests" for Servlet and Reactive applications.

Servlet applications

Applications need to configure the org.springframework.web.filter.ServerHttpObservationFilter Servlet filter in their application. It is using the org.springframework.http.server.observation.DefaultServerRequestObservationConvention by default, backed by the ServerRequestObservationContext.

This will only record an observation as an error if the Exception has not been handled by the web Framework and has bubbled up to the Servlet filter. Typically, all exceptions handled by Spring MVC’s @ExceptionHandler and ProblemDetail support will not be recorded with the observation. You can, at any point during request processing, set the error field on the ObservationContext yourself:

code:UserController

By default, the following KeyValues are created:

Table 1. Low cardinality Keys

Name	Description
exception (required)	Name of the exception thrown during the exchange, or KeyValue#NONE_VALUE} if no exception happened.
method (required)	Name of HTTP request method or "none" if the request was not received properly.
outcome (required)	Outcome of the HTTP server exchange.
status (required)	HTTP response raw status code, or "UNKNOWN" if no response was created.
uri (required)	URI pattern for the matching handler if available, falling back to REDIRECTION for 3xx responses, NOT_FOUND for 404 responses, root for requests with no path info, and UNKNOWN for all other requests.

Table 2. High cardinality Keys

Name	Description
http.url (required)	HTTP request URI.

Reactive applications

Applications need to configure the org.springframework.web.filter.reactive.ServerHttpObservationFilter reactive WebFilter in their application. It is using the org.springframework.http.server.reactive.observation.DefaultServerRequestObservationConvention by default, backed by the ServerRequestObservationContext.

This will only record an observation as an error if the Exception has not been handled by the web Framework and has bubbled up to the WebFilter. Typically, all exceptions handled by Spring WebFlux’s @ExceptionHandler and ProblemDetail support will not be recorded with the observation. You can, at any point during request processing, set the error field on the ObservationContext yourself:

code:UserController

By default, the following KeyValues are created:

Table 3. Low cardinality Keys

Name	Description
exception (required)	Name of the exception thrown during the exchange, or "none" if no exception happened.
method (required)	Name of HTTP request method or "none" if the request was not received properly.
outcome (required)	Outcome of the HTTP server exchange.
status (required)	HTTP response raw status code, or "UNKNOWN" if no response was created.
uri (required)	URI pattern for the matching handler if available, falling back to REDIRECTION for 3xx responses, NOT_FOUND for 404 responses, root for requests with no path info, and UNKNOWN for all other requests.

Table 4. High cardinality Keys

Name	Description
http.url (required)	HTTP request URI.

HTTP Client instrumentation

HTTP client exchanges observations are created with the name "http.client.requests" for blocking and reactive clients. Unlike their server counterparts, the instrumentation is implemented directly in the client so the only required step is to configure an ObservationRegistry on the client.

RestTemplate

Applications must configure an ObservationRegistry on RestTemplate instances to enable the instrumentation; without that, observations are "no-ops". Spring Boot will auto-configure RestTemplateBuilder beans with the observation registry already set.

Instrumentation is using the org.springframework.http.client.observation.ClientRequestObservationConvention by default, backed by the ClientRequestObservationContext.

Table 5. Low cardinality Keys

Name	Description
method (required)	Name of HTTP request method or "none" if the request could not be created.
uri (required)	URI template used for HTTP request, or "none" if none was provided. Only the path part of the URI is considered.
client.name (required)	Client name derived from the request URI host.
status (required)	HTTP response raw status code, or "IO_ERROR" in case of IOException, or "CLIENT_ERROR" if no response was received.
outcome (required)	Outcome of the HTTP client exchange.
exception (required)	Name of the exception thrown during the exchange, or "none" if no exception happened.

Table 6. High cardinality Keys

Name	Description
http.url (required)	HTTP request URI.

WebClient

Applications must configure an ObservationRegistry on the WebClient builder to enable the instrumentation; without that, observations are "no-ops". Spring Boot will auto-configure WebClient.Builder beans with the observation registry already set.

Instrumentation is using the org.springframework.web.reactive.function.client.ClientRequestObservationConvention by default, backed by the ClientRequestObservationContext.

Table 7. Low cardinality Keys

Name	Description
method (required)	Name of HTTP request method or "none" if the request could not be created.
uri (required)	URI template used for HTTP request, or "none" if none was provided. Only the path part of the URI is considered.
client.name (required)	Client name derived from the request URI host.
status (required)	HTTP response raw status code, or "IO_ERROR" in case of IOException, or "CLIENT_ERROR" if no response was received.
outcome (required)	Outcome of the HTTP client exchange.
exception (required)	Name of the exception thrown during the exchange, or "none" if no exception happened.

Table 8. High cardinality Keys

Name	Description
http.url (required)	HTTP request URI.