Skip to main content
The Metrics Explorer charts the metrics your services send. Pick a metric from the catalog, choose how to aggregate it, group it by labels, and click into the exemplars behind a histogram.
The Metrics Explorer with a line chart of http.server.request.duration p95 and the series table below it

The Metrics Explorer charting the p95 of http.server.request.duration

​
Build a query

Each row of the explorer is one query.
  1. Metric. Pick a name from the catalog. The picker shows the unit next to the name. When a metric declares no unit, Sherlock infers seconds, milliseconds, or bytes from a name ending in _seconds, _milliseconds, or _bytes, with or without a following _total. Names ending in _timestamp_seconds, _start_time_seconds, or _boot_time_seconds stay unitless unless a unit is declared. The catalog refreshes every 60 seconds.
  2. Aggregation. The choices depend on the metric type. A counter offers sum, avg, min, max, rate, and count, with rate as the default. A gauge offers last, avg, min, max, sum, and count, with last as the default; a gauge is a level, so it has no rate. A histogram offers p50, p90, p95, p99, avg, min, max, sum, and count, with p95 as the default; min and max use delta data only, so a histogram sent as cumulative returns no values for them. A summary offers avg, sum, count, and rate, with avg as the default. A cumulative UpDownCounter is listed as a gauge and takes the gauge aggregations; see Metric temporality.
  3. Group by. Pick a label to get one series per value, or No grouping for a single series. The list holds the metric’s own attributes and its resource attributes, for example a node, pod, namespace, or host name. When both carry the key, the data point’s value is used.
  4. Filter. Add label filters to narrow the series. Filters take the same attributes as Group by, so a resource attribute such as k8s.namespace.name works as a filter.
Add metric adds another row. Bands replaces a histogram row with four rows, p50, p90, p95, and p99, so the tail and the median share one chart.

​
Read the chart

  • Line, Bars, Area, Stacked, and Table show any query. Heatmap and Dist need a single histogram query and show the bucket distribution over time or over the whole window.
  • lin and log switch the value axis.
  • vs prev period overlays the same query shifted back by the selected range.
  • Threshold draws a horizontal line at a value you choose.
  • The header shows the step and the window the chart was computed with, for example step 30s · 1h window.
  • View SQL shows the query that produced the chart. The SQL tab runs your own query against the metrics tables.
  • Share copies a link that reproduces the whole explorer state.
The series table under the chart lists every series with its value over the window, n ≥ p for percentile queries, the change versus the previous period, and a sparkline. Click the eye icon to hide a series.

​
Exemplars

A histogram carries exemplars: sampled measurements with the trace id of the request that produced them.
  • Click a series row in the table to open the exemplars of that series in the window. Results are limited, so the drawer may not include every recorded exemplar.
  • In Heatmap, click a cell to open the exemplars of that bucket and time step.
  • In Dist, click a bar to open the exemplars of that value range.
The Exemplars drawer lists them with time, duration, and trace id. View trace opens the trace. Exemplars walks through the whole pivot.

​
Before data arrives

An empty Metrics page shows four setup steps: the OTLP metrics endpoint, the bearer token in the Authorization header, instrumenting the service with the env resource attribute, and emitting at least one data point. The catalog refreshes every 60 seconds.

​
Troubleshooting

Counters and histograms chart whether they arrive as delta or cumulative. A cumulative point is charted as the increase over its predecessor, and Sherlock looks for that predecessor from ten minutes before the window starts. A point with none is omitted, not plotted as zero, so a new cumulative series charts from its second point. Delta points chart from the first. See Metric temporality.
If totals are about twice as high, check for attributes that separate duplicate scrapes of the same target, such as each collector’s own pod name, and for two collectors converting the same series to delta with the cumulativetodelta processor. Totals that jump can mean different processes share one series identity; give each one an identity such as service.instance.id. See Running more than one collector.
Prometheus summaries chart count, sum, avg, and rate. Their precomputed quantiles are not offered. If you need percentiles you can aggregate across instances, expose the value as a histogram. See What Sherlock charts.

Exemplars

From a spike to the request and its logs.

Custom metrics

Define your own histograms with exemplars in Node.js.