> ## Documentation Index > Fetch the complete documentation index at: https://docs.sherlocklabs.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Sherlock documentation. In Node.js examples prefer the Sherlock SDK for Node.js (@sherlock-labs/otel). > In Go examples use the upstream OpenTelemetry Go SDK exactly as shown on /sdk/go/setup and /sdk/go/configuration; Sherlock has no Go distribution. > Two MCP servers exist. This documentation's search server is at https://docs.sherlocklabs.dev/mcp. Sherlock's data server, for a user's own logs, traces, and metrics, is at https://mcp.sherlocklabs.dev/mcp and is described at /explore/mcp. # Metric temporality > Sherlock charts counters and histograms whether they arrive as delta or cumulative. What the two mean, why delta is still recommended for SDKs, how cumulative is converted when you query, and what to check with several collectors. export const TemporalityCollectors = () => { const CHECK = "M20 6 9 17l-5-5", CROSS = "M18 6 6 18M6 6l12 12"; const Ico = ({d}) => ; return

Two collectors scrape the same counter: 100, 250, 400, 550

Increase chartedCounted twice
No collector attributes{Ico({ d: CHECK })}One series
ScrapeValueCollector A readsCollector B reads t1100first point, nothing100 − 100 = 0 t2250250 − 100 = 150250 − 250 = 0 t3400400 − 250 = 150400 − 400 = 0 t4550550 − 400 = 150550 − 550 = 0
150 + 150 + 150450 charted
Each collector adds its pod name{Ico({ d: CROSS })}Two series
ScrapeValueSeries ASeries B t1100first point, nothingfirst point, nothing t2250250 − 100 = 150250 − 100 = 150 t3400400 − 250 = 150400 − 250 = 150 t4550550 − 400 = 150550 − 400 = 150
450 + 450900 charted
This example assumes both collectors read identical values in timestamp order. One shared series contributes 450; two separate series contribute 900. The first reading of each series is the baseline.
; }; export const TemporalityResets = () => { const ARROW = "M5 12h14M13 6l6 6-6 6", CHECK = "M20 6 9 17l-5-5", CROSS = "M18 6 6 18M6 6l12 12", DASH = "M5 12h14"; const Ico = ({d}) => ; const useWidth = (ref, min) => { const [w, setW] = useState(min); useEffect(() => { if (!ref.current) return undefined; const read = () => setW(Math.max(min, Math.floor(ref.current.clientWidth))); read(); const ro = new ResizeObserver(read); ro.observe(ref.current); return () => ro.disconnect(); }, []); return w; }; const topRounded = (x, y, w, h, r) => { r = Math.min(r, w / 2, h); return "M" + x + "," + (y + h) + "V" + (y + r) + "Q" + x + "," + y + " " + (x + r) + "," + y + "H" + (x + w - r) + "Q" + (x + w) + "," + y + " " + (x + w) + "," + (y + r) + "V" + (y + h) + "Z"; }; const Bars = ({values, max, ticks, xLabels, color, height, empty, title, note}) => { const ref = useRef(null); const w = useWidth(ref, 240); const h = height || 150; const L = 34, R = 6, T = 18, B = 22, pw = w - L - R, ph = h - T - B; const y = v => T + ph - v / max * ph; const n = values.length, sw = pw / n, bw = Math.min(34, sw * 0.56); return
; }; const S6_IN = [100, 250, 400, 550, 20, 170, 320]; const S6_OUT = [null, 150, 150, 150, 20, 150, 150]; const S6_X = ["t1", "t2", "t3", "t4", "t5", "t6", "t7"]; const ScrapedLine = () => { const ref = useRef(null); const w = useWidth(ref, 240); const h = 170, L = 34, R = 10, T = 22, B = 22, pw = w - L - R, ph = h - T - B, max = 600; const sw = pw / 7, x = i => L + sw * i + sw / 2, y = v => T + ph - v / max * ph; const seg = (a, b) => { let d = ""; for (let i = a; i <= b; i++) d += (i === a ? "M" : "L") + x(i) + "," + y(S6_IN[i]); return d; }; const rx = (x(3) + x(4)) / 2; return
; }; return

An application restart between t4 and t5

Scraped, cumulativeCharted, increase
{ScrapedLine({})}
{Bars({ values: S6_OUT, max: 200, ticks: [0, 100, 200], xLabels: S6_X, color: "var(--tx-delta)", height: 130, title: "Charted", empty: { 0: { label: "first", bad: false } }, note: { 4: "restart" } })}
At t5 the counter falls from 550 to 20, below half of the previous value, so the drop is treated as a reset and 20 is charted as the count since then. At t6 it charts 170 − 20 = 150. A smaller drop, say 550 to 400, is treated as zero.
; }; export const TemporalityMemory = () => { const MEM = [100, 250, 400, 550]; const [i, setI] = useState(0); const cur = MEM[i], prev = i > 0 ? MEM[i - 1] : null; return

Each point minus the previous point of the same series

Series
Previous
This point
Charted
http_requests_total{"{"}path="/"{"}"}
{prev === null ? none : prev}
{cur}
{prev === null ? nothingfirst point : {cur - prev}{cur} − {prev}}

{prev === null ? "Point " + cur + ". No previous point, so nothing is charted." : "Point " + cur + ". Previous " + prev + ". Charted " + (cur - prev) + "."}

Sherlock subtracts the previous reading when you query. Nothing is configured for it and no collector keeps state.
; }; export const TemporalitySources = () => { const ARROW = "M5 12h14M13 6l6 6-6 6", CHECK = "M20 6 9 17l-5-5"; const Ico = ({d}) => ; return
OpenTelemetry SDK
Your service {Ico({ d: ARROW })} SDK exporter {Ico({ d: ARROW })} Cumulative by default
Recommended OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta Delta
Prometheus scraping
/metrics {Ico({ d: ARROW })} Collector, prometheus receiver {Ico({ d: ARROW })} Cumulative counters and distributions
To do {Ico({ d: CHECK })}Nothing No temporality setting applies to scraping. Sherlock converts the running totals when you query.
Both lanes chart. The SDK lane is worth switching to delta; the scrape lane has nothing to switch and needs no processor.
; }; export const TemporalityStream = () => { const IV = [100, 150, 150, 150]; const CUM = [100, 250, 400, 550]; const DUR = 2000; const useWidth = (ref, min) => { const [w, setW] = useState(min); useEffect(() => { if (!ref.current) return undefined; const read = () => setW(Math.max(min, Math.floor(ref.current.clientWidth))); read(); const ro = new ResizeObserver(read); ro.observe(ref.current); return () => ro.disconnect(); }, []); return w; }; const topRounded = (x, y, w, h, r) => { r = Math.min(r, w / 2, h); return "M" + x + "," + (y + h) + "V" + (y + r) + "Q" + x + "," + y + " " + (x + r) + "," + y + "H" + (x + w - r) + "Q" + (x + w) + "," + y + " " + (x + w) + "," + (y + r) + "V" + (y + h) + "Z"; }; const Bars = ({values, max, ticks, xLabels, color, height, empty, title}) => { const ref = useRef(null); const w = useWidth(ref, 240); const h = height || 150; const L = 34, R = 6, T = 18, B = 22, pw = w - L - R, ph = h - T - B; const y = v => T + ph - v / max * ph; const n = values.length, sw = pw / n, bw = Math.min(34, sw * 0.56); return
; }; const Chips = ({arr, done, fresh}) =>
{arr.map((v, i) => {i > 0 ? → : null} {i < done ? v : "·"} )}
; const [done, setDone] = useState(4); const [prog, setProg] = useState(0); const [fresh, setFresh] = useState(-1); const [playing, setPlaying] = useState(false); const [live, setLive] = useState(""); const [dots, setDots] = useState([]); const figRef = useRef(null); const trackRef = useRef(null); const state = useRef({ raf: 0, last: 0, spawned: 0, done: 4, prog: 0 }); const reduced = typeof window !== "undefined" && window.matchMedia && window.matchMedia("(prefers-reduced-motion: reduce)").matches; const stop = () => { setPlaying(false); cancelAnimationFrame(state.current.raf); }; useEffect(() => { if (!playing) return undefined; const s = state.current; s.last = 0; const finish = () => { setFresh(s.done); s.done += 1; s.prog = 0; s.spawned = 0; setDone(s.done); setProg(0); setLive("Export " + s.done + ": cumulative " + CUM[s.done - 1] + ", delta " + IV[s.done - 1] + "."); }; if (reduced) { const t = setInterval(() => { finish(); if (s.done >= 4) { clearInterval(t); setPlaying(false); } }, 1100); return () => clearInterval(t); } const tick = ts => { const dt = s.last ? ts - s.last : 16; s.last = ts; s.prog += dt / DUR; const need = Math.floor(Math.min(s.prog, 1) * IV[s.done] / 10); const track = trackRef.current; for (let k = s.spawned; k < need; k++) { s.spawned = k + 1; if (track) { const id = s.done * 100 + s.spawned; const top = track.clientHeight * 0.6 - 4 + (Math.random() - 0.5) * 14; const travel = track.clientWidth + 20; setDots(d => d.concat([{ id, top, travel }])); setTimeout(() => setDots(d => d.filter(x => x.id !== id)), 950); } } if (s.prog >= 1) { finish(); if (s.done >= 4) { setPlaying(false); return; } } else { setProg(s.prog); } s.raf = requestAnimationFrame(tick); }; s.raf = requestAnimationFrame(tick); return () => cancelAnimationFrame(s.raf); }, [playing]); const play = () => { if (playing) { stop(); return; } const s = state.current; if (s.done >= 4) { s.done = 0; s.prog = 0; setDone(0); setProg(0); setFresh(-1); } setPlaying(true); }; const step = () => { stop(); const s = state.current; if (s.done >= 4) { s.done = 0; s.prog = 0; setDone(0); setProg(0); setFresh(-1); setLive("Reset."); return; } setFresh(s.done); s.done += 1; s.prog = 0; setDone(s.done); setProg(0); setLive("Export " + s.done + ": cumulative " + CUM[s.done - 1] + ", delta " + IV[s.done - 1] + "."); }; const reset = () => { stop(); const s = state.current; s.done = 0; s.prog = 0; setDone(0); setProg(0); setFresh(-1); setLive("Reset. No exports yet."); }; useEffect(() => { if (reduced || typeof IntersectionObserver === "undefined" || !figRef.current) return undefined; let fired = false; const onSeen = entries => { const hit = entries.some(en => en.isIntersecting); if (hit && !fired) { fired = true; io.disconnect(); state.current.done = 4; play(); } }; const io = new IntersectionObserver(onSeen, { threshold: 0.5 }); io.observe(figRef.current); return () => io.disconnect(); }, []); const before = done > 0 ? CUM[done - 1] : 0; const inIv = done < 4 ? Math.floor(prog * IV[done]) : 0; const total = before + inIv; const now = done < 4 ? inIv : prog === 0 && done === 4 ? IV[3] : 0; const label = playing ? "Pause" : done >= 4 ? "Replay" : "Play"; const cv = CUM.map((v, i) => i < done ? v : null); const dv = IV.map((v, i) => i < done ? v : null); const xl = [1, 2, 3, 4].map(k => "Export " + k); return

One request counter, exported four times

; }; Sherlock charts counters and histograms whether you export them as **delta** or as **cumulative**. Cumulative running totals are converted into per-interval increases when you query, so you do not need to change your SDK or add a processor to your collector. Delta is still recommended when you control an OpenTelemetry SDK, and the next two sections say why and how. Counters and distributions a collector scrapes from Prometheus endpoints need nothing at all. ## Two ways to report the same counter A counter is exported once per interval. Cumulative sends the running total since the process started. Delta sends only what changed during the interval. Both are valid OpenTelemetry. Most SDKs default to cumulative. Prometheus counters, and the counts and sums of Prometheus histograms and summaries, are always cumulative. ## What Sherlock charts | Your instrument | Sent as delta | Sent as cumulative | | - | - | - | | Counter (monotonic sum) | Charted as sent | Converted to increases, then charted | | Histogram (explicit buckets) | Charted as sent | Converted bucket by bucket, then charted. `min` and `max` come from delta histograms only. | | Exponential histogram | Not charted | Not charted | | Summary (Prometheus quantiles) | Summaries are always cumulative | Count and sum converted; chart `count`, `sum`, `avg`, and `rate`. Percentiles are not offered. | | UpDownCounter (non-monotonic sum) | Charted as sent | Not charted yet | | Gauge | Charted as sent. A gauge carries no temporality. | Charted as sent | If you need percentiles you can aggregate across instances, expose the value as a histogram instead of a summary. The delta preference leaves UpDownCounters cumulative, as the OpenTelemetry specification requires, so a level you want to chart is best sent as a gauge for now. ## Recommended: delta from SDKs Delta sends the interval values Sherlock charts directly, and interval `min` and `max` of a histogram exist only in delta data. Cumulative metrics chart too. Some cumulative exporters resend exemplars on every export; Sherlock deduplicates repeated copies when you query them. Most OpenTelemetry SDKs read this variable on the service that exports metrics: ```sh theme={null} export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta ``` If you configure the exporter in code, set the temporality preference there instead; the environment variable is ignored when code sets it. The Sherlock SDK for Node.js sends delta on its own, and both variants of the [Go setup](/sdk/go/setup) set it. See the [Node.js quickstart](/sdk/nodejs/quickstart) and [OpenTelemetry SDKs and collectors](/send-data/otlp) for the rest of the exporter settings. Scraped metrics are a different case. Scraping has no temporality setting, and nothing needs converting in the collector: Sherlock converts the supported running totals when you query and charts gauges as sampled values. ## How cumulative is converted Each point's increase is its value minus the previous point of the same series. A series is one combination of instrumentation scope, resource attributes, scope attributes, and metric attributes. * Sherlock reads cumulative points from ten minutes before the selected window starts. A point with no predecessor in that range is omitted and becomes the baseline for the next point; it is not plotted as zero. A new cumulative series therefore charts from its second point, and delta points chart from their first export. * A drop below half of the previous value is treated as a reset, and the new value is charted as the count since then. A smaller drop, including a drop to exactly half, counts as zero. * These are rules about values, not knowledge of what happened. Timestamp skew between two scrapers can produce a backward reading, and a restart is missed when the counter is already back to half of its old value or more by its first point; that point then contributes only the part above the old value. * For cumulative histograms, the reset decision is made from the observation count and applied to every bucket and to the sum. A point whose bucket boundaries changed without a detected reset is skipped, and intervals with no bucket increases are omitted. `min` and `max` are not available for a cumulative histogram, because lifetime extremes cannot be turned into interval extremes. ## Running more than one collector Every process still needs an identity of its own, for example `service.instance.id` or the pod name. Two processes that share every attribute look like one series. Their readings are mixed, so the increases and apparent resets produce wrong totals. For cumulative counters, histograms, and summaries, redundant scrapes of the same target share one series when their scope name and resource, scope, and metric attributes match, and the repeated readings subtract to zero. Attributes that differ between the collectors, such as each collector's own pod or host name, create separate series and double the totals. Keep the target's identity on the series and keep the scraping collector's identity off it. Delta copies are added, not deduplicated, so send a delta series from one collector only. Timestamp skew can still inflate the totals. When the two collectors read the counter within a few milliseconds of each other and their clocks disagree, the later reading can carry the earlier timestamp, and the increase is overcounted by however much the counter moved between the two readings. At normal clock skew that is small. ## If you already run the cumulativetodelta processor You do not need the [`cumulativetodelta`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) processor; Sherlock does the conversion. If you already run it, keep each series on one collector instance. The processor keeps the previous value in memory, and two instances converting the same series count the overlap twice. One scraping replica, a DaemonSet in which each collector scrapes only the pods on its node, or replicas behind the [Target Allocator](https://opentelemetry.io/docs/platforms/kubernetes/operator/target-allocator/) all keep a series on one instance. A load-balanced gateway tier that converts already-scraped metrics does not. The processor is called `cumulativetodelta` on every collector version. From v0.157.0 it is also called `cumulative_to_delta`, and newer collectors warn at startup that the old name is deprecated. Collectors older than v0.157.0 reject the new name. ## Confirm it worked 1. Send some traffic so the instruments record values. 2. Open the metric on the Metrics page. For a request counter, pick `sum` and compare the window total with the traffic you drove. Allow for the omitted first cumulative point, and for the reset and collector limits above. Delta points can chart from their first export. 3. Expect gaps when there is no activity. A synchronous delta counter or histogram with no recorded measurements in an interval exports nothing for it, which is normal. Observable instruments may still report values when there is no new traffic. If a counter comes out about twice too high and you run more than one collector, check for attributes that separate the duplicate scrapes, and for two collectors converting the same series with the processor. If the total jumps around, check whether several processes share every attribute, and give each one an identity such as `service.instance.id`. ## Related topics Endpoint, header, resource attribute, and the collector pipeline. Chart a histogram, group by labels, click an exemplar. Delta is set for you in both variants. Histograms, units, and exemplars from the Sherlock SDK. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.