Go configuration

This page assumes the sherlock.go and main.go from Setup, in either variant. Each section adds one thing to them. Every snippet below compiles against the pinned versions on that page.

Environment variables

Variable	Default	Meaning
`SHERLOCK_ACCESS_TOKEN`	none	Sent as `Authorization: Bearer <token>` on every export. Required.
`SHERLOCK_ENDPOINT`	none	The ingest URL from Settings → Collector. Required. `sherlock.go` appends `/v1/traces` and `/v1/metrics`. An `http://` scheme switches both exporters to plaintext, which is useful against a local collector.
`OTEL_SERVICE_NAME`	none	`service.name`. Required.
`OTEL_RESOURCE_ATTRIBUTES`	none	`env=<value>` routes the data into a source. Add more pairs with commas: `env=prod,service.version=1.4.2`.
`OTEL_METRIC_EXPORT_INTERVAL`	`60000`	Metrics export interval in milliseconds. The Sherlock SDK for Node.js uses 30 seconds; set `30000` to match.
`OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`	`parentbased_always_on`	Sampler and ratio. See Sampling.

OTEL_METRICS_EXEMPLAR_FILTER has no effect with the settings-in-the-file variant: it sets the filter in code, and code wins over the environment. The environment variant on the Setup page reads it.

Skip health checks

There is no default ignore list. A filtered request produces neither a span nor a data point, so a health checker that hits /healthz every few seconds does not show up anywhere.

handler := otelhttp.NewHandler(mux, "http.server",
	otelhttp.WithFilter(func(r *http.Request) bool {
		switch r.URL.Path {
		case "/healthz", "/livez", "/healthcheck":
			return false
		}
		return true
	}))

Outbound calls made while handling a filtered request are still traced; they just have no parent.

Custom histograms

Record durations in seconds with unit s, and pass Sherlock’s boundaries so the chart matches the HTTP histograms. durationBoundariesS is the list in sherlock.go.

meter := otel.Meter("app")
workDuration, err := meter.Float64Histogram("app.work.duration",
	metric.WithUnit("s"),
	metric.WithDescription("time spent on one job"),
	metric.WithExplicitBucketBoundaries(durationBoundariesS...))
if err != nil {
	log.Fatal(err)
}

// In a handler. The context carries the request's span, so the exemplar
// gets trace_id, span_id, and trace_flags without any extra code.
started := time.Now()
doWork()
workDuration.Record(r.Context(), time.Since(started).Seconds(),
	metric.WithAttributes(attribute.String("kind", "report")))

Every recording is a candidate exemplar; the reservoir keeps one per bucket per export. Unsampled requests get exemplars too, with trace_flags 00.

Every attribute you pass to Record becomes a metric label. A job id or a user id there creates one series per value. Put such values on the exemplar only, as shown next.

Exemplar-only attributes

To carry a high-cardinality value on the exemplar but not on the data point, record it as a normal attribute and drop it from the stream with an attribute filter. The SDK moves dropped attributes onto the exemplar. Add the case to sherlockView in sherlock.go; do not add a second view, because two views that match one instrument export it twice.

	switch i.Name {
	case "http.server.request.duration", "http.client.request.duration":
		s.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{
			Boundaries: durationBoundariesS,
		}
	case "app.work.duration":
		s.AttributeFilter = attribute.NewDenyKeysFilter("jobId")
	}

workDuration.Record(r.Context(), seconds, metric.WithAttributes(
	attribute.String("kind", "report"),
	attribute.String("jobId", jobID), // exemplar only, see below
))

A typo in the filter key silently turns the value into a metric label. Check the series on the Metrics page after the first export.

Outbound HTTP

Wrap the transport. Each call gets a client span, W3C trace context headers for the callee, and http.client.request.duration with the same 20 boundaries.

client := &http.Client{
	Transport: otelhttp.NewTransport(http.DefaultTransport),
}
req, _ := http.NewRequestWithContext(
	r.Context(), http.MethodGet, url, nil)
resp, err := client.Do(req)

Build requests with the incoming request’s context. A request built with context.Background() starts a new trace.

Manual spans

tracer := otel.Tracer("app")

ctx, span := tracer.Start(r.Context(), "render report")
defer span.End()
span.SetAttributes(attribute.String("report.kind", "monthly"))

Pass ctx to everything the span covers, including Record calls and log lines, so they attach to it.

http.route on spans

otelhttp names the span after the mux pattern and puts http.route on the metric, but it does not set http.route on the span. If you filter traces by route, add this middleware inside otelhttp.NewHandler and outside the mux. It runs after routing, while the span is still open.

func withRoute(next http.Handler) http.Handler {
	return http.HandlerFunc(func(
		w http.ResponseWriter, r *http.Request,
	) {
		next.ServeHTTP(w, r)
		if i := strings.IndexByte(r.Pattern, '/'); i >= 0 {
			span := trace.SpanFromContext(r.Context())
			span.SetAttributes(semconv.HTTPRoute(r.Pattern[i:]))
		}
	})
}

// in main
handler := otelhttp.NewHandler(withRoute(mux), "http.server")

semconv is go.opentelemetry.io/otel/semconv/v1.37.0.

Sampling

The default samples every trace. To sample a fraction, set the sampler in the environment:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

or in code, which overrides the environment:

tracerProvider := sdktrace.NewTracerProvider(
	sdktrace.WithResource(res),
	sdktrace.WithBatcher(traceExporter),
	sdktrace.WithSampler(
		sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))))

ParentBased follows the caller’s decision when a request arrives with trace context, so a trace is sampled whole across services. Metrics count every request regardless. Unsampled requests still produce exemplars, marked trace_flags 00; those exemplars have no trace to open.

Kill switches

Go has no built-in way to turn a signal off at runtime. Two small wrappers give you one. Both keep a flag you set from wherever you like, an admin endpoint or a config watcher. Traces. A sampler wrapper. Wrapped in ParentBased, it stops new traces only; requests already being traced finish whole.

var tracesEnabled atomic.Bool // set true at start

type toggleSampler struct{ inner sdktrace.Sampler }

func (s toggleSampler) ShouldSample(
	p sdktrace.SamplingParameters,
) sdktrace.SamplingResult {
	if !tracesEnabled.Load() {
		parent := trace.SpanContextFromContext(p.ParentContext)
		return sdktrace.SamplingResult{
			Decision:   sdktrace.Drop,
			Tracestate: parent.TraceState(),
		}
	}
	return s.inner.ShouldSample(p)
}

func (s toggleSampler) Description() string {
	return "toggle(" + s.inner.Description() + ")"
}

// in Setup
sdktrace.WithSampler(sdktrace.ParentBased(
	toggleSampler{sdktrace.TraceIDRatioBased(1)}))

Metrics. An exporter wrapper that drops whole batches. With delta temporality a disabled window is a gap, never a double count.

var metricsEnabled atomic.Bool // set true at start

type toggleExporter struct{ sdkmetric.Exporter }

func (e toggleExporter) Export(
	ctx context.Context, rm *metricdata.ResourceMetrics,
) error {
	if !metricsEnabled.Load() {
		return nil
	}
	return e.Exporter.Export(ctx, rm)
}

// in Setup
sdkmetric.WithReader(
	sdkmetric.NewPeriodicReader(toggleExporter{metricExporter}))

metricdata is go.opentelemetry.io/otel/sdk/metric/metricdata.

Logs with trace ids

Neither setup sends logs on its own. Two ways get log lines next to their trace in Sherlock. Both need context-aware calls: slog.InfoContext(r.Context(), "work done") gets the ids, slog.Info("work done") does not.

Send logs over OTLP

Four more modules, all on the pre-stable logs line of the SDK:

go get go.opentelemetry.io/otel/log@v0.22.0 \
  go.opentelemetry.io/otel/sdk/log@v0.22.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp@v0.22.0 \
  go.opentelemetry.io/contrib/bridges/otelslog@v0.20.1

Add this to Setup in sherlock.go, after the meter provider. It uses the same endpoint, headers, and res. The otelslog handler turns every slog record into an OTLP log record, and the SDK stamps trace_id, span_id, and trace_flags from the context.

logExp, err := otlploghttp.New(ctx,
	otlploghttp.WithEndpointURL(endpoint+"/v1/logs"),
	otlploghttp.WithHeaders(headers))
if err != nil {
	return nil, err
}
loggerProvider := sdklog.NewLoggerProvider(
	sdklog.WithResource(res),
	sdklog.WithProcessor(sdklog.NewBatchProcessor(logExp)))
logHandler := otelslog.NewHandler("app",
	otelslog.WithLoggerProvider(loggerProvider))
slog.SetDefault(slog.New(logHandler))

Then flush and stop it with the others in the returned function:

return errors.Join(
	tracerProvider.ForceFlush(ctx), meterProvider.ForceFlush(ctx),
	loggerProvider.ForceFlush(ctx),
	tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx),
	loggerProvider.Shutdown(ctx))

sdklog is go.opentelemetry.io/otel/sdk/log. The attributes of a slog call become the log’s attributes as they are, so an error logged as slog.ErrorContext(ctx, "boom", "err", err) shows up on the Errors page when the source’s error rule reads LogAttributes['err']. Lines written through the standard log package after slog.SetDefault are exported too, without trace ids. To keep JSON on stdout as well, fan the record out to two handlers. Handle may modify the record, so give each handler its own r.Clone(), and forward WithAttrs and WithGroup so logger.With(...) keeps the fan-out.

Stamp the ids and ship with a collector

If a collector already reads your stdout or files, keep logging JSON and add the ids yourself. This slog.Handler wrapper needs only the trace package. The WithAttrs and WithGroup methods matter: without them the wrapper is lost on the first logger.With(...).

type traceHandler struct{ slog.Handler }

func (h traceHandler) Handle(
	ctx context.Context, r slog.Record,
) error {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(
			slog.String("trace_id", sc.TraceID().String()),
			slog.String("span_id", sc.SpanID().String()),
			slog.String("trace_flags", sc.TraceFlags().String()))
	}
	return h.Handler.Handle(ctx, r)
}

func (h traceHandler) WithAttrs(as []slog.Attr) slog.Handler {
	return traceHandler{h.Handler.WithAttrs(as)}
}

func (h traceHandler) WithGroup(n string) slog.Handler {
	return traceHandler{h.Handler.WithGroup(n)}
}

// at start
handler := traceHandler{slog.NewJSONHandler(os.Stdout, nil)}
slog.SetDefault(slog.New(handler))

Send the lines with an OpenTelemetry Collector, as on OpenTelemetry SDKs and collectors. The trace_id field is the same lowercase hex Sherlock stores for the span, so the Logs page links each line to its trace.

Histogram boundaries

The settings-in-the-file variant of sherlock.go gives the two HTTP duration histograms Sherlock’s 20 boundaries in seconds. The environment variant keeps otelhttp’s 14, which skip the steps between 100 and 500 ms, so p95 is coarser there. To add the 20 to the environment variant, replace its traceFlagsOnExemplars with this version, which is the other variant’s sherlockView under another name:

// traceFlagsOnExemplars is a view over every histogram that adds trace_flags,
// "01" sampled or "00" not, to each exemplar. OTLP has no field for the flag,
// so this is the one part of the setup that cannot come from the environment.
func traceFlagsOnExemplars(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) {
	if inst.Kind != sdkmetric.InstrumentKindHistogram {
		return sdkmetric.Stream{}, false
	}
	stream := sdkmetric.Stream{
		Name:                              inst.Name,
		Description:                       inst.Description,
		Unit:                              inst.Unit,
		ExemplarReservoirProviderSelector: reservoirsWithTraceFlags,
	}
	switch inst.Name {
	case "http.server.request.duration", "http.client.request.duration":
		stream.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{
			Boundaries: []float64{
				0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3,
				0.35, 0.4, 0.45, 0.5, 0.75, 1, 2.5, 5, 7.5, 10,
			},
		}
	}
	return stream, true
}

Runtime metrics

The settings-in-the-file variant calls runtime.Start, which gives seven go.* metrics. With the environment variant, add go.opentelemetry.io/contrib/instrumentation/runtime@v0.71.0 to the modules and start it after the meter provider is set:

if err := runtime.Start(runtime.WithMeterProvider(meterProvider)); err != nil {
	return nil, err
}

Scheduler latency, go.schedule.duration, needs a producer on the reader in either variant:

sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter,
	sdkmetric.WithProducer(runtime.NewProducer())))

Baggage

The environment variant propagates trace context only. To forward baggage as well to the services you call:

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
	propagation.TraceContext{}, propagation.Baggage{}))

Shutdown

The order in the main.go from Setup matters: stop the HTTP server first, so in-flight requests finish and record their spans and durations, then call the function Setup returned. It flushes both providers and shuts them down under the deadline you pass. Both exports land, including the last one.

Endpoint on a private network

To send through a proxy that needs a different Host header, WithHeaders does not work: Go’s HTTP client takes the authority from the request, not the header map. Give the exporter a client whose transport sets req.Host.

type hostRewriter struct {
	host string
	base http.RoundTripper
}

func (h hostRewriter) RoundTrip(
	r *http.Request,
) (*http.Response, error) {
	r = r.Clone(r.Context())
	r.Host = h.host
	return h.base.RoundTrip(r)
}

// on each exporter
otlpmetrichttp.WithHTTPClient(&http.Client{
	Transport: hostRewriter{
		host: "ingest.internal", base: http.DefaultTransport,
	},
	Timeout:   10 * time.Second,
})

WithHTTPClient replaces the exporter’s own client, so WithTimeout, WithProxy, and WithTLSClientConfig no longer apply. Set them on your client.

Routers and nested muxes

The tested path is net/http with the Go 1.22 ServeMux, where otelhttp.NewHandler reads the route from the mux pattern. Muxes mounted with http.StripPrefix report the mount pattern, not the inner route. StripPrefix hands the inner mux a copy of the request, so the inner pattern never reaches the request otelhttp holds. A service with mux.Handle("/api/v1/auth/", http.StripPrefix("/api/v1/auth", authMux)) gets one span name and one http.route, POST /api/v1/auth/, for every route inside authMux. To keep the inner routes apart, register full patterns on one mux, or mount the inner mux without StripPrefix and register its patterns with the prefix. Other routers need their own contrib middleware to get a route: otelgin for Gin, otelecho for Echo, otelmux for gorilla/mux, and otelchi for chi. sherlockView applies to any histogram named http.server.request.duration, whichever middleware produces it. Check what your middleware emits on the Metrics page after the first export; span-only middlewares leave you without the request duration histogram.

Differences from Node.js

Exemplar counts, ignore lists, kill switches, and other traps.

Metrics

Chart a histogram, group by labels, click an exemplar.

Logs

Search log lines and open the trace behind one.

OpenTelemetry SDKs and collectors

Ship logs from files or stdout with a collector.

Send data

Node.js SDK

Go SDK

Environment variables

Skip health checks

Custom histograms

Exemplar-only attributes

Outbound HTTP

Manual spans

http.route on spans

Sampling

Kill switches

Logs with trace ids

Send logs over OTLP

Stamp the ids and ship with a collector

Histogram boundaries

Runtime metrics

Baggage

Shutdown

Endpoint on a private network

Routers and nested muxes

Differences from Node.js

Metrics

Logs

OpenTelemetry SDKs and collectors

​Environment variables

​Skip health checks

​Custom histograms

​Exemplar-only attributes

​Outbound HTTP

​Manual spans

​http.route on spans

​Sampling

​Kill switches

​Logs with trace ids

​Send logs over OTLP

​Stamp the ids and ship with a collector

​Histogram boundaries

​Runtime metrics

​Baggage

​Shutdown

​Endpoint on a private network

​Routers and nested muxes

​Related topics

Differences from Node.js

Metrics

Logs

OpenTelemetry SDKs and collectors

Environment variables

Skip health checks

Custom histograms

Exemplar-only attributes

Outbound HTTP

Manual spans

http.route on spans

Sampling

Kill switches

Logs with trace ids

Send logs over OTLP

Stamp the ids and ship with a collector

Histogram boundaries

Runtime metrics

Baggage

Shutdown

Endpoint on a private network

Routers and nested muxes

Related topics