> ## Documentation Index > Fetch the complete documentation index at: https://docs.sherlocklabs.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Sherlock documentation. In Node.js examples prefer the Sherlock SDK for Node.js (@sherlock-labs/otel). > In Go examples use the upstream OpenTelemetry Go SDK exactly as shown on /sdk/go/setup and /sdk/go/configuration; Sherlock has no Go distribution. > Two MCP servers exist. This documentation's search server is at https://sherlock-c8721ead.mintlify.app/mcp. Sherlock's data server, for a user's own logs, traces, and metrics, is at https://mcp.sherlocklabs.dev/mcp and is described at /explore/mcp. # Go configuration > Production settings for the upstream OpenTelemetry Go SDK with Sherlock: health checks, custom histograms, exemplar-only attributes, sampling, kill switches, logs, runtime metrics, and shutdown. This page assumes the `sherlock.go` and `main.go` from [Setup](/sdk/go/setup), in either variant. Each section adds one thing to them. Every snippet below compiles against the pinned versions on that page. ## Environment variables | Variable | Default | Meaning | | ------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `SHERLOCK_ACCESS_TOKEN` | none | Sent as `Authorization: Bearer ` on every export. Required. | | `SHERLOCK_ENDPOINT` | none | The ingest URL from **Settings → Collector**. Required. `sherlock.go` appends `/v1/traces` and `/v1/metrics`. An `http://` scheme switches both exporters to plaintext, which is useful against a local collector. | | `OTEL_SERVICE_NAME` | none | `service.name`. Required. | | `OTEL_RESOURCE_ATTRIBUTES` | none | `env=` routes the data into a source. Add more pairs with commas: `env=prod,service.version=1.4.2`. | | `OTEL_METRIC_EXPORT_INTERVAL` | `60000` | Metrics export interval in milliseconds. The Sherlock SDK for Node.js uses 30 seconds; set `30000` to match. | | `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | `parentbased_always_on` | Sampler and ratio. See [Sampling](#sampling). | `OTEL_METRICS_EXEMPLAR_FILTER` has no effect with the settings-in-the-file variant: it sets the filter in code, and code wins over the environment. The environment variant on the [Setup](/sdk/go/setup) page reads it. ## Skip health checks There is no default ignore list. A filtered request produces neither a span nor a data point, so a health checker that hits `/healthz` every few seconds does not show up anywhere. ```go theme={null} handler := otelhttp.NewHandler(mux, "http.server", otelhttp.WithFilter(func(r *http.Request) bool { switch r.URL.Path { case "/healthz", "/livez", "/healthcheck": return false } return true })) ``` Outbound calls made while handling a filtered request are still traced; they just have no parent. ## Custom histograms Record durations in seconds with unit `s`, and pass Sherlock's boundaries so the chart matches the HTTP histograms. `durationBoundariesS` is the list in `sherlock.go`. ```go theme={null} meter := otel.Meter("app") workDuration, err := meter.Float64Histogram("app.work.duration", metric.WithUnit("s"), metric.WithDescription("time spent on one job"), metric.WithExplicitBucketBoundaries(durationBoundariesS...)) if err != nil { log.Fatal(err) } // In a handler. The context carries the request's span, so the exemplar // gets trace_id, span_id, and trace_flags without any extra code. started := time.Now() doWork() workDuration.Record(r.Context(), time.Since(started).Seconds(), metric.WithAttributes(attribute.String("kind", "report"))) ``` Every recording is a candidate exemplar; the reservoir keeps one per bucket per export. Unsampled requests get exemplars too, with `trace_flags` `00`. Every attribute you pass to `Record` becomes a metric label. A job id or a user id there creates one series per value. Put such values on the exemplar only, as shown next. ### Exemplar-only attributes To carry a high-cardinality value on the exemplar but not on the data point, record it as a normal attribute and drop it from the stream with an attribute filter. The SDK moves dropped attributes onto the exemplar. Add the case to `sherlockView` in `sherlock.go`; do not add a second view, because two views that match one instrument export it twice. ```go theme={null} switch i.Name { case "http.server.request.duration", "http.client.request.duration": s.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{ Boundaries: durationBoundariesS, } case "app.work.duration": s.AttributeFilter = attribute.NewDenyKeysFilter("jobId") } ``` ```go theme={null} workDuration.Record(r.Context(), seconds, metric.WithAttributes( attribute.String("kind", "report"), attribute.String("jobId", jobID), // exemplar only, see below )) ``` A typo in the filter key silently turns the value into a metric label. Check the series on the Metrics page after the first export. ## Outbound HTTP Wrap the transport. Each call gets a client span, W3C trace context headers for the callee, and `http.client.request.duration` with the same 20 boundaries. ```go theme={null} client := &http.Client{ Transport: otelhttp.NewTransport(http.DefaultTransport), } req, _ := http.NewRequestWithContext( r.Context(), http.MethodGet, url, nil) resp, err := client.Do(req) ``` Build requests with the incoming request's context. A request built with `context.Background()` starts a new trace. ## Manual spans ```go theme={null} tracer := otel.Tracer("app") ctx, span := tracer.Start(r.Context(), "render report") defer span.End() span.SetAttributes(attribute.String("report.kind", "monthly")) ``` Pass `ctx` to everything the span covers, including `Record` calls and log lines, so they attach to it. ## http.route on spans `otelhttp` names the span after the mux pattern and puts `http.route` on the metric, but it does not set `http.route` on the span. If you filter traces by route, add this middleware inside `otelhttp.NewHandler` and outside the mux. It runs after routing, while the span is still open. ```go theme={null} func withRoute(next http.Handler) http.Handler { return http.HandlerFunc(func( w http.ResponseWriter, r *http.Request, ) { next.ServeHTTP(w, r) if i := strings.IndexByte(r.Pattern, '/'); i >= 0 { span := trace.SpanFromContext(r.Context()) span.SetAttributes(semconv.HTTPRoute(r.Pattern[i:])) } }) } // in main handler := otelhttp.NewHandler(withRoute(mux), "http.server") ``` `semconv` is `go.opentelemetry.io/otel/semconv/v1.37.0`. ## Sampling The default samples every trace. To sample a fraction, set the sampler in the environment: ```sh theme={null} export OTEL_TRACES_SAMPLER=parentbased_traceidratio export OTEL_TRACES_SAMPLER_ARG=0.1 ``` or in code, which overrides the environment: ```go theme={null} tracerProvider := sdktrace.NewTracerProvider( sdktrace.WithResource(res), sdktrace.WithBatcher(traceExporter), sdktrace.WithSampler( sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))) ``` `ParentBased` follows the caller's decision when a request arrives with trace context, so a trace is sampled whole across services. Metrics count every request regardless. Unsampled requests still produce exemplars, marked `trace_flags` `00`; those exemplars have no trace to open. ## Kill switches Go has no built-in way to turn a signal off at runtime. Two small wrappers give you one. Both keep a flag you set from wherever you like, an admin endpoint or a config watcher. **Traces.** A sampler wrapper. Wrapped in `ParentBased`, it stops new traces only; requests already being traced finish whole. ```go theme={null} var tracesEnabled atomic.Bool // set true at start type toggleSampler struct{ inner sdktrace.Sampler } func (s toggleSampler) ShouldSample( p sdktrace.SamplingParameters, ) sdktrace.SamplingResult { if !tracesEnabled.Load() { parent := trace.SpanContextFromContext(p.ParentContext) return sdktrace.SamplingResult{ Decision: sdktrace.Drop, Tracestate: parent.TraceState(), } } return s.inner.ShouldSample(p) } func (s toggleSampler) Description() string { return "toggle(" + s.inner.Description() + ")" } // in Setup sdktrace.WithSampler(sdktrace.ParentBased( toggleSampler{sdktrace.TraceIDRatioBased(1)})) ``` **Metrics.** An exporter wrapper that drops whole batches. With delta temporality a disabled window is a gap, never a double count. ```go theme={null} var metricsEnabled atomic.Bool // set true at start type toggleExporter struct{ sdkmetric.Exporter } func (e toggleExporter) Export( ctx context.Context, rm *metricdata.ResourceMetrics, ) error { if !metricsEnabled.Load() { return nil } return e.Exporter.Export(ctx, rm) } // in Setup sdkmetric.WithReader( sdkmetric.NewPeriodicReader(toggleExporter{metricExporter})) ``` `metricdata` is `go.opentelemetry.io/otel/sdk/metric/metricdata`. ## Logs with trace ids Neither setup sends logs on its own. Two ways get log lines next to their trace in Sherlock. Both need context-aware calls: `slog.InfoContext(r.Context(), "work done")` gets the ids, `slog.Info("work done")` does not. ### Send logs over OTLP Four more modules, all on the pre-stable logs line of the SDK: ```sh theme={null} go get go.opentelemetry.io/otel/log@v0.22.0 \ go.opentelemetry.io/otel/sdk/log@v0.22.0 \ go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp@v0.22.0 \ go.opentelemetry.io/contrib/bridges/otelslog@v0.20.1 ``` Add this to `Setup` in `sherlock.go`, after the meter provider. It uses the same `endpoint`, `headers`, and `res`. The `otelslog` handler turns every `slog` record into an OTLP log record, and the SDK stamps `trace_id`, `span_id`, and `trace_flags` from the context. ```go theme={null} logExp, err := otlploghttp.New(ctx, otlploghttp.WithEndpointURL(endpoint+"/v1/logs"), otlploghttp.WithHeaders(headers)) if err != nil { return nil, err } loggerProvider := sdklog.NewLoggerProvider( sdklog.WithResource(res), sdklog.WithProcessor(sdklog.NewBatchProcessor(logExp))) logHandler := otelslog.NewHandler("app", otelslog.WithLoggerProvider(loggerProvider)) slog.SetDefault(slog.New(logHandler)) ``` Then flush and stop it with the others in the returned function: ```go theme={null} return errors.Join( tracerProvider.ForceFlush(ctx), meterProvider.ForceFlush(ctx), loggerProvider.ForceFlush(ctx), tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx), loggerProvider.Shutdown(ctx)) ``` `sdklog` is `go.opentelemetry.io/otel/sdk/log`. The attributes of a `slog` call become the log's attributes as they are, so an error logged as `slog.ErrorContext(ctx, "boom", "err", err)` shows up on the Errors page when the source's error rule reads `LogAttributes['err']`. Lines written through the standard `log` package after `slog.SetDefault` are exported too, without trace ids. To keep JSON on stdout as well, fan the record out to two handlers. `Handle` may modify the record, so give each handler its own `r.Clone()`, and forward `WithAttrs` and `WithGroup` so `logger.With(...)` keeps the fan-out. ### Stamp the ids and ship with a collector If a collector already reads your stdout or files, keep logging JSON and add the ids yourself. This `slog.Handler` wrapper needs only the `trace` package. The `WithAttrs` and `WithGroup` methods matter: without them the wrapper is lost on the first `logger.With(...)`. ```go theme={null} type traceHandler struct{ slog.Handler } func (h traceHandler) Handle( ctx context.Context, r slog.Record, ) error { if sc := trace.SpanContextFromContext(ctx); sc.IsValid() { r.AddAttrs( slog.String("trace_id", sc.TraceID().String()), slog.String("span_id", sc.SpanID().String()), slog.String("trace_flags", sc.TraceFlags().String())) } return h.Handler.Handle(ctx, r) } func (h traceHandler) WithAttrs(as []slog.Attr) slog.Handler { return traceHandler{h.Handler.WithAttrs(as)} } func (h traceHandler) WithGroup(n string) slog.Handler { return traceHandler{h.Handler.WithGroup(n)} } // at start handler := traceHandler{slog.NewJSONHandler(os.Stdout, nil)} slog.SetDefault(slog.New(handler)) ``` Send the lines with an OpenTelemetry Collector, as on [OpenTelemetry SDKs and collectors](/send-data/otlp#logs). The `trace_id` field is the same lowercase hex Sherlock stores for the span, so the Logs page links each line to its trace. ## Histogram boundaries The settings-in-the-file variant of `sherlock.go` gives the two HTTP duration histograms Sherlock's 20 boundaries in seconds. The environment variant keeps otelhttp's 14, which skip the steps between 100 and 500 ms, so p95 is coarser there. To add the 20 to the environment variant, replace its `traceFlagsOnExemplars` with this version, which is the other variant's `sherlockView` under another name: ```go theme={null} // traceFlagsOnExemplars is a view over every histogram that adds trace_flags, // "01" sampled or "00" not, to each exemplar. OTLP has no field for the flag, // so this is the one part of the setup that cannot come from the environment. func traceFlagsOnExemplars(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) { if inst.Kind != sdkmetric.InstrumentKindHistogram { return sdkmetric.Stream{}, false } stream := sdkmetric.Stream{ Name: inst.Name, Description: inst.Description, Unit: inst.Unit, ExemplarReservoirProviderSelector: reservoirsWithTraceFlags, } switch inst.Name { case "http.server.request.duration", "http.client.request.duration": stream.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{ Boundaries: []float64{ 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.75, 1, 2.5, 5, 7.5, 10, }, } } return stream, true } ``` ## Runtime metrics The settings-in-the-file variant calls `runtime.Start`, which gives seven `go.*` metrics. With the environment variant, add `go.opentelemetry.io/contrib/instrumentation/runtime@v0.71.0` to the modules and start it after the meter provider is set: ```go theme={null} if err := runtime.Start(runtime.WithMeterProvider(meterProvider)); err != nil { return nil, err } ``` Scheduler latency, `go.schedule.duration`, needs a producer on the reader in either variant: ```go theme={null} sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter, sdkmetric.WithProducer(runtime.NewProducer()))) ``` ## Baggage The environment variant propagates trace context only. To forward baggage as well to the services you call: ```go theme={null} otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{})) ``` ## Shutdown The order in the `main.go` from [Setup](/sdk/go/setup) matters: stop the HTTP server first, so in-flight requests finish and record their spans and durations, then call the function `Setup` returned. It flushes both providers and shuts them down under the deadline you pass. Both exports land, including the last one. ## Endpoint on a private network To send through a proxy that needs a different `Host` header, `WithHeaders` does not work: Go's HTTP client takes the authority from the request, not the header map. Give the exporter a client whose transport sets `req.Host`. ```go theme={null} type hostRewriter struct { host string base http.RoundTripper } func (h hostRewriter) RoundTrip( r *http.Request, ) (*http.Response, error) { r = r.Clone(r.Context()) r.Host = h.host return h.base.RoundTrip(r) } // on each exporter otlpmetrichttp.WithHTTPClient(&http.Client{ Transport: hostRewriter{ host: "ingest.internal", base: http.DefaultTransport, }, Timeout: 10 * time.Second, }) ``` `WithHTTPClient` replaces the exporter's own client, so `WithTimeout`, `WithProxy`, and `WithTLSClientConfig` no longer apply. Set them on your client. ## Routers and nested muxes The tested path is `net/http` with the Go 1.22 `ServeMux`, where `otelhttp.NewHandler` reads the route from the mux pattern. **Muxes mounted with `http.StripPrefix` report the mount pattern, not the inner route.** `StripPrefix` hands the inner mux a copy of the request, so the inner pattern never reaches the request `otelhttp` holds. A service with `mux.Handle("/api/v1/auth/", http.StripPrefix("/api/v1/auth", authMux))` gets one span name and one `http.route`, `POST /api/v1/auth/`, for every route inside `authMux`. To keep the inner routes apart, register full patterns on one mux, or mount the inner mux without `StripPrefix` and register its patterns with the prefix. Other routers need their own contrib middleware to get a route: `otelgin` for Gin, `otelecho` for Echo, `otelmux` for gorilla/mux, and `otelchi` for chi. `sherlockView` applies to any histogram named `http.server.request.duration`, whichever middleware produces it. Check what your middleware emits on the Metrics page after the first export; span-only middlewares leave you without the request duration histogram. ## Related topics Exemplar counts, ignore lists, kill switches, and other traps. Chart a histogram, group by labels, click an exemplar. Search log lines and open the trace behind one. Ship logs from files or stdout with a collector.