Skip to main content
Sherlock charts counters and histograms whether you export them as delta or as cumulative. Cumulative running totals are converted into per-interval increases when you query, so you do not need to change your SDK or add a processor to your collector. Delta is still recommended when you control an OpenTelemetry SDK, and the next two sections say why and how. Counters and distributions a collector scrapes from Prometheus endpoints need nothing at all.

​
Two ways to report the same counter

A counter is exported once per interval. Cumulative sends the running total since the process started. Delta sends only what changed during the interval. Both are valid OpenTelemetry. Most SDKs default to cumulative. Prometheus counters, and the counts and sums of Prometheus histograms and summaries, are always cumulative.

​
What Sherlock charts

If you need percentiles you can aggregate across instances, expose the value as a histogram instead of a summary. The delta preference leaves UpDownCounters cumulative, as the OpenTelemetry specification requires, so a level you want to chart is best sent as a gauge for now. Delta sends the interval values Sherlock charts directly, and interval min and max of a histogram exist only in delta data. Cumulative metrics chart too. Some cumulative exporters resend exemplars on every export; Sherlock deduplicates repeated copies when you query them. Most OpenTelemetry SDKs read this variable on the service that exports metrics:
If you configure the exporter in code, set the temporality preference there instead; the environment variable is ignored when code sets it. The Sherlock SDK for Node.js sends delta on its own, and both variants of the Go setup set it. See the Node.js quickstart and OpenTelemetry SDKs and collectors for the rest of the exporter settings. Scraped metrics are a different case. Scraping has no temporality setting, and nothing needs converting in the collector: Sherlock converts the supported running totals when you query and charts gauges as sampled values.

​
How cumulative is converted

Each point’s increase is its value minus the previous point of the same series. A series is one combination of instrumentation scope, resource attributes, scope attributes, and metric attributes.
  • Sherlock reads cumulative points from ten minutes before the selected window starts. A point with no predecessor in that range is omitted and becomes the baseline for the next point; it is not plotted as zero. A new cumulative series therefore charts from its second point, and delta points chart from their first export.
  • A drop below half of the previous value is treated as a reset, and the new value is charted as the count since then. A smaller drop, including a drop to exactly half, counts as zero.
  • These are rules about values, not knowledge of what happened. Timestamp skew between two scrapers can produce a backward reading, and a restart is missed when the counter is already back to half of its old value or more by its first point; that point then contributes only the part above the old value.
  • For cumulative histograms, the reset decision is made from the observation count and applied to every bucket and to the sum. A point whose bucket boundaries changed without a detected reset is skipped, and intervals with no bucket increases are omitted. min and max are not available for a cumulative histogram, because lifetime extremes cannot be turned into interval extremes.

​
Running more than one collector

Every process still needs an identity of its own, for example service.instance.id or the pod name. Two processes that share every attribute look like one series. Their readings are mixed, so the increases and apparent resets produce wrong totals. For cumulative counters, histograms, and summaries, redundant scrapes of the same target share one series when their scope name and resource, scope, and metric attributes match, and the repeated readings subtract to zero. Attributes that differ between the collectors, such as each collector’s own pod or host name, create separate series and double the totals. Keep the target’s identity on the series and keep the scraping collector’s identity off it. Delta copies are added, not deduplicated, so send a delta series from one collector only. Timestamp skew can still inflate the totals. When the two collectors read the counter within a few milliseconds of each other and their clocks disagree, the later reading can carry the earlier timestamp, and the increase is overcounted by however much the counter moved between the two readings. At normal clock skew that is small.

​
If you already run the cumulativetodelta processor

You do not need the cumulativetodelta processor; Sherlock does the conversion. If you already run it, keep each series on one collector instance. The processor keeps the previous value in memory, and two instances converting the same series count the overlap twice. One scraping replica, a DaemonSet in which each collector scrapes only the pods on its node, or replicas behind the Target Allocator all keep a series on one instance. A load-balanced gateway tier that converts already-scraped metrics does not. The processor is called cumulativetodelta on every collector version. From v0.157.0 it is also called cumulative_to_delta, and newer collectors warn at startup that the old name is deprecated. Collectors older than v0.157.0 reject the new name.

​
Confirm it worked

  1. Send some traffic so the instruments record values.
  2. Open the metric on the Metrics page. For a request counter, pick sum and compare the window total with the traffic you drove. Allow for the omitted first cumulative point, and for the reset and collector limits above. Delta points can chart from their first export.
  3. Expect gaps when there is no activity. A synchronous delta counter or histogram with no recorded measurements in an interval exports nothing for it, which is normal. Observable instruments may still report values when there is no new traffic.
If a counter comes out about twice too high and you run more than one collector, check for attributes that separate the duplicate scrapes, and for two collectors converting the same series with the processor. If the total jumps around, check whether several processes share every attribute, and give each one an identity such as service.instance.id.

OpenTelemetry SDKs and collectors

Endpoint, header, resource attribute, and the collector pipeline.

Metrics

Chart a histogram, group by labels, click an exemplar.

Go setup

Delta is set for you in both variants.

Custom metrics in Node.js

Histograms, units, and exemplars from the Sherlock SDK.