> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sherlocklabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Sherlock documentation. In Node.js examples prefer the Sherlock SDK for Node.js (@sherlock-labs/otel).
> In Go examples use the upstream OpenTelemetry Go SDK exactly as shown on /sdk/go/setup and /sdk/go/configuration; Sherlock has no Go distribution.
> Two MCP servers exist. This documentation's search server is at https://sherlock-c8721ead.mintlify.app/mcp. Sherlock's data server, for a user's own logs, traces, and metrics, is at https://mcp.sherlocklabs.dev/mcp and is described at /explore/mcp.
# Go setup
> Send traces and metrics from a Go service to Sherlock with the upstream OpenTelemetry Go SDK: one bootstrap file, with the Sherlock settings either in the file or in the standard OTEL_* environment variables.
Sherlock has no Go distribution and needs none. The upstream OpenTelemetry Go SDK sends what Sherlock expects once four things are set: delta metrics, exemplars on every request, a `trace_flags` attribute on each exemplar, and Sherlock's histogram boundaries. One file, `sherlock.go`, takes care of them. This page adds it to a `net/http` service.
The file comes in two variants that send the same data. In the first, every Sherlock setting is written in the file, so a reader sees all of them in one place. In the second, the standard `OTEL_*` environment variables carry the settings, which suits a team that already configures OpenTelemetry that way, and the file is shorter. Step 2 has both.
[Configuration](/sdk/go/configuration) covers production settings such as health checks, custom histograms, sampling, and logs. [Differences from Node.js](/sdk/go/differences) lists what the Go setup does differently from the Sherlock SDK for Node.js.
## What you'll learn
* Which modules to add, at which versions
* The two places the Sherlock settings can live, and what stays in code either way
* What `sherlock.go` sets, and why each setting is there
* Where the first traces and metrics show up
## Prerequisites
* Go 1.22 or later, for `ServeMux` method-and-path patterns. Tested with Go 1.25.
* A Sherlock organization and its bearer token from **Settings → Collector**. [Get started](/get-started/getting-started) shows where.
* A service on `net/http`. For other routers, see [Routers and nested muxes](/sdk/go/configuration#routers-and-nested-muxes).
## Steps
```sh theme={null}
go get go.opentelemetry.io/otel@v1.46.0 \
go.opentelemetry.io/otel/sdk@v1.46.0 \
go.opentelemetry.io/otel/sdk/metric@v1.46.0 \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.46.0 \
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp@v1.46.0 \
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.71.0 \
go.opentelemetry.io/contrib/instrumentation/runtime@v0.71.0
```
These are the versions this page was tested with. Recent contrib releases changed behavior that this page relies on: `otelhttp.WithRouteTag` was removed and the `OTEL_SEMCONV_STABILITY_OPT_IN` switch is gone. Keep the pins until you have tested an upgrade.
Pick one variant. Save the file next to your `main.go` and export the variables it expects. Nothing in either file is specific to your routes.
```go sherlock.go theme={null}
// sherlock.go: OpenTelemetry setup for a Go service that sends to Sherlock.
package main
import (
"context"
"errors"
"os"
"time"
"go.opentelemetry.io/contrib/instrumentation/runtime"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/metric/exemplar"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/trace"
)
// Setup starts the traces and metrics pipelines. Call it once at start-up.
// Call the function it returns after your HTTP server has stopped; it
// flushes what is still buffered.
func Setup(ctx context.Context) (func(context.Context) error, error) {
// Both values are on the Sherlock Settings → Collector page.
endpoint := os.Getenv("SHERLOCK_ENDPOINT")
token := os.Getenv("SHERLOCK_ACCESS_TOKEN")
if endpoint == "" || token == "" {
return nil, errors.New("set SHERLOCK_ENDPOINT and SHERLOCK_ACCESS_TOKEN")
}
headers := map[string]string{"Authorization": "Bearer " + token}
// service.name and env come from OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES.
res, err := resource.New(ctx, resource.WithTelemetrySDK(), resource.WithFromEnv())
if err != nil {
return nil, err
}
traceExporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpointURL(endpoint+"/v1/traces"),
otlptracehttp.WithHeaders(headers))
if err != nil {
return nil, err
}
tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithResource(res),
sdktrace.WithBatcher(traceExporter))
metricExporter, err := otlpmetrichttp.New(ctx,
otlpmetrichttp.WithEndpointURL(endpoint+"/v1/metrics"),
otlpmetrichttp.WithHeaders(headers),
// Sherlock stores delta metrics. The SDK default is cumulative.
otlpmetrichttp.WithTemporalitySelector(sdkmetric.DeltaTemporalitySelector))
if err != nil {
return nil, err
}
meterProvider := sdkmetric.NewMeterProvider(
sdkmetric.WithResource(res),
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
// Exemplars on every request, not only on sampled ones.
sdkmetric.WithExemplarFilter(exemplar.AlwaysOnFilter),
sdkmetric.WithView(sherlockView))
otel.SetTracerProvider(tracerProvider)
otel.SetMeterProvider(meterProvider)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
if err := runtime.Start(runtime.WithMeterProvider(meterProvider)); err != nil {
return nil, err
}
return func(ctx context.Context) error {
return errors.Join(
tracerProvider.ForceFlush(ctx), meterProvider.ForceFlush(ctx),
tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx))
}, nil
}
// durationBoundariesS is the bucket list Sherlock charts durations on, in seconds.
var durationBoundariesS = []float64{
0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.25, 0.3,
0.35, 0.4, 0.45, 0.5, 0.75, 1, 2.5, 5, 7.5, 10,
}
// sherlockView is the one view for every histogram: exemplars get trace_flags,
// and the two HTTP duration histograms get Sherlock's boundaries. Keep it one
// view; two views that match the same instrument export it twice.
func sherlockView(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) {
if inst.Kind != sdkmetric.InstrumentKindHistogram {
return sdkmetric.Stream{}, false
}
stream := sdkmetric.Stream{
Name: inst.Name,
Description: inst.Description,
Unit: inst.Unit,
ExemplarReservoirProviderSelector: reservoirsWithTraceFlags,
}
switch inst.Name {
case "http.server.request.duration", "http.client.request.duration":
stream.Aggregation = sdkmetric.AggregationExplicitBucketHistogram{
Boundaries: durationBoundariesS,
}
}
return stream, true
}
// OTLP exemplars carry trace_id and span_id but not the sampled flag. The
// reservoir wrapper below adds trace_flags, "01" sampled or "00" not, to each
// exemplar, which tells Sherlock whether there is a trace to open.
func reservoirsWithTraceFlags(agg sdkmetric.Aggregation) exemplar.ReservoirProvider {
provider := sdkmetric.DefaultExemplarReservoirProviderSelector(agg)
return func(attrs attribute.Set) exemplar.Reservoir {
return flagged{provider(attrs)}
}
}
type flagged struct{ exemplar.Reservoir }
func (f flagged) Offer(
ctx context.Context, t time.Time, v exemplar.Value, attrs []attribute.KeyValue,
) {
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
flags := attribute.String("trace_flags", sc.TraceFlags().String())
attrs = append(attrs[:len(attrs):len(attrs)], flags)
}
f.Reservoir.Offer(ctx, t, v, attrs)
}
```
What it sets, and why:
| Setting | Why |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WithTemporalitySelector(DeltaTemporalitySelector)` | Sherlock stores delta metrics. The SDK default is cumulative, which also re-exports stale exemplars. |
| `WithExemplarFilter(exemplar.AlwaysOnFilter)` | The default `trace_based` filter drops exemplars for unsampled requests. Sherlock wants them, with `trace_flags` `00`. |
| `reservoirsWithTraceFlags` | OTLP exemplars carry `trace_id` and `span_id` but not the sampled flag. The wrapper adds `trace_flags` as a filtered attribute, so the UI knows which exemplars open a trace. |
| `sherlockView` | One view over every histogram. It injects the reservoir and gives the two HTTP duration histograms Sherlock's 20 boundaries in seconds. It is one function on purpose: two views that match the same instrument export it twice. |
| `resource.WithFromEnv()` | Reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`, so the service name and the `env` value stay out of the code. |
| `runtime.Start` | Seven `go.*` runtime metrics. |
The file reads four variables:
```sh theme={null}
export SHERLOCK_ENDPOINT='' # Settings → Collector
export SHERLOCK_ACCESS_TOKEN='' # Settings → Collector, Reveal
export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=env=prod # picks the source
```
**Settings → Collector** shows the Endpoint and the Bearer Token for your organization; copy both, since the endpoint differs by organization. `env` selects the source the data lands in, so use the value you chose in [Get started](/get-started/getting-started#step-3-choose-the-env-value). A value that matches no source is not shown.
```go sherlock.go theme={null}
// sherlock.go: OpenTelemetry setup for Sherlock, configured through the
// standard OTEL_* environment variables. Only trace_flags needs code.
package main
import (
"context"
"errors"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/metric/exemplar"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/trace"
)
// Setup starts the traces and metrics pipelines. The exporters read the
// endpoint, the headers, and the temporality from the environment; the
// providers read the service name, the resource attributes, and the
// exemplar filter. Call the returned function after your server has stopped.
func Setup(ctx context.Context) (func(context.Context) error, error) {
traceExporter, err := otlptracehttp.New(ctx)
if err != nil {
return nil, err
}
metricExporter, err := otlpmetrichttp.New(ctx)
if err != nil {
return nil, err
}
tracerProvider := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExporter))
meterProvider := sdkmetric.NewMeterProvider(
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
sdkmetric.WithView(traceFlagsOnExemplars))
otel.SetTracerProvider(tracerProvider)
otel.SetMeterProvider(meterProvider)
otel.SetTextMapPropagator(propagation.TraceContext{})
return func(ctx context.Context) error {
return errors.Join(tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx))
}, nil
}
// traceFlagsOnExemplars is a view over every histogram that adds trace_flags,
// "01" sampled or "00" not, to each exemplar. OTLP has no field for the flag,
// so this is the one part of the setup that cannot come from the environment.
func traceFlagsOnExemplars(inst sdkmetric.Instrument) (sdkmetric.Stream, bool) {
if inst.Kind != sdkmetric.InstrumentKindHistogram {
return sdkmetric.Stream{}, false
}
return sdkmetric.Stream{
Name: inst.Name,
Description: inst.Description,
Unit: inst.Unit,
ExemplarReservoirProviderSelector: reservoirsWithTraceFlags,
}, true
}
func reservoirsWithTraceFlags(agg sdkmetric.Aggregation) exemplar.ReservoirProvider {
provider := sdkmetric.DefaultExemplarReservoirProviderSelector(agg)
return func(attrs attribute.Set) exemplar.Reservoir {
return flagged{provider(attrs)}
}
}
type flagged struct{ exemplar.Reservoir }
func (f flagged) Offer(
ctx context.Context, t time.Time, v exemplar.Value, attrs []attribute.KeyValue,
) {
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
flags := attribute.String("trace_flags", sc.TraceFlags().String())
attrs = append(attrs[:len(attrs):len(attrs)], flags)
}
f.Reservoir.Offer(ctx, t, v, attrs)
}
```
The exporters and providers take no options. Everything they need is in these six variables:
```sh theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT='' # Settings → Collector
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer '
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
export OTEL_METRICS_EXEMPLAR_FILTER=always_on
export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=env=prod
```
| Variable | What it does |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | The Endpoint from **Settings → Collector**. The exporters append `/v1/traces` and `/v1/metrics`. Without it they send to `localhost:4318`. |
| `OTEL_EXPORTER_OTLP_HEADERS` | Headers as `key=value` pairs, comma-separated. Replace `` with the Bearer Token from the same page. |
| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `delta`. Sherlock stores delta metrics; the SDK default is cumulative. |
| `OTEL_METRICS_EXEMPLAR_FILTER` | `always_on`, so unsampled requests get exemplars too, marked `trace_flags` `00`. |
| `OTEL_SERVICE_NAME` | `service.name`. |
| `OTEL_RESOURCE_ATTRIBUTES` | `env=` routes the data into a source. See [Get started](/get-started/getting-started#step-3-choose-the-env-value). |
What stays in code is `trace_flags` on exemplars: OTLP has no field for the sampled flag, so no variable can add it.
This variant keeps the SDK defaults where the other one sets Sherlock's values: otelhttp's 14 histogram boundaries instead of Sherlock's 20, no runtime metrics, and trace-context propagation without baggage. Each comes back with a few lines; see [Histogram boundaries](/sdk/go/configuration#histogram-boundaries), [Runtime metrics](/sdk/go/configuration#runtime-metrics), and [Baggage](/sdk/go/configuration#baggage).
Two mistakes stay quiet with this variant. With the endpoint variable unset, the SDK sends to `localhost:4318` and prints this to stderr on every export:
```text theme={null}
traces export: Post "https://localhost:4318/v1/traces":
dial tcp [::1]:4318: connect: connection refused
```
With the temporality variable unset there is no message at all: the data arrives cumulative and every chart climbs forever. Check the variables before the first run, and the Metrics page after it.
Call `Setup` before the server starts, wrap the mux with `otelhttp.NewHandler`, and on shutdown stop the server first, then flush telemetry. This `main.go` is complete and works with either variant.
```go main.go theme={null}
// main.go: a net/http service, instrumented by Setup from sherlock.go.
package main
import (
"context"
"log"
"net/http"
"os/signal"
"syscall"
"time"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM)
defer stop()
shutdown, err := Setup(ctx)
if err != nil {
log.Fatal(err)
}
mux := http.NewServeMux()
mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(`{"hello":"world"}`))
})
mux.HandleFunc("GET /work", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(150 * time.Millisecond)
w.Write([]byte(`{"done":true}`))
})
// otelhttp.NewHandler wraps the mux, so every route is traced and metered.
srv := &http.Server{
Addr: ":8080",
Handler: otelhttp.NewHandler(mux, "http.server"),
ReadHeaderTimeout: 5 * time.Second,
}
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatal(err)
}
}()
log.Print("listening on :8080")
<-ctx.Done()
closeCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
// Finish in-flight requests first, then flush the last spans and metrics.
if err := srv.Shutdown(closeCtx); err != nil {
log.Print(err)
}
if err := shutdown(closeCtx); err != nil {
log.Print(err)
}
}
```
Register handlers with method-and-path patterns, such as `GET /work`. The span name and the `http.route` metric attribute come from the pattern. A handler registered as a bare path gives a span named `GET` with no route, and a mux mounted behind `http.StripPrefix` reports its mount pattern for every inner route. See [Routers and nested muxes](/sdk/go/configuration#routers-and-nested-muxes).
```sh theme={null}
go run .
```
Send a few requests so there is something to look at:
```sh theme={null}
for i in 1 2 3 4 5; do
curl -s localhost:8080/work
curl -s localhost:8080/hello
done
```
* **Traces.** Open **Traces**, switch to the **Spans** view, and pick `checkout-api`. Spans leave in batches of a few seconds. Each request from this `main.go` is one span named by its pattern, `GET /work`, with `http.response.status_code` on it. The **Traces** view lists only traces with more than one span, so it stays empty until a request makes a nested call.
* **Metrics.** Open **Metrics** and choose `http.server.request.duration`. It appears after the first export, 60 seconds by default. Group by `http.route` to see the two routes apart. Each bucket carries an exemplar; click one to open the request's trace.
* **Runtime.** With the settings-in-the-file variant, `go.goroutine.count` and `go.memory.used` are on the same page.
Stop the app with Ctrl-C. The shutdown waits for the server to close, then flushes the last spans and metrics; both arrive.
## What arrives
| Data | Name | Unit |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| Server spans | `GET /work`, `GET /hello`, kind server, scope `otelhttp` | |
| Request duration | `http.server.request.duration`, delta, 20 boundaries (14 with the environment variant) | `s` |
| Body sizes | `http.server.request.body.size`, `http.server.response.body.size` | `By` |
| Runtime | `go.goroutine.count`, `go.memory.used`, `go.memory.allocated`, `go.memory.allocations`, `go.memory.gc.goal`, `go.config.gogc`, `go.processor.limit` (settings-in-the-file variant) | various |
| Exemplars | one per non-empty histogram bucket per export, with `trace_id`, `span_id`, `trace_flags` | |
Outbound calls through an `otelhttp.NewTransport` client add `http.client.request.duration` with the same boundaries. Logs are not sent by this setup. [Logs with trace ids](/sdk/go/configuration#logs-with-trace-ids) adds them over OTLP with the same bearer token, or stamps stdout lines for a collector.
## Troubleshooting
Export errors go to stderr through the SDK's error handler, prefixed with `traces export:` or `failed to upload metrics:`. A `401` means the token is wrong or missing. Check that the endpoint and token variables are set in the shell that runs the service, that both match **Settings → Collector**, and that the endpoint has no trailing path.
The `env` value in `OTEL_RESOURCE_ATTRIBUTES` did not match a source. Compare it with the source's match value under **Settings → Collector**.
A request matched no pattern, or a health checker hit the service. Requests that match no pattern get no `http.route`. Health checks need a filter; there is no default ignore list. See [Skip health checks](/sdk/go/configuration#skip-health-checks).
The periodic reader exports every 60 seconds by default. Set `OTEL_METRIC_EXPORT_INTERVAL=30000` for 30 seconds. The catalog refreshes every 60 seconds on top of that.
## Related topics
Health checks, custom histograms, sampling, kill switches, logs, shutdown.
What to expect if you know the Sherlock SDK for Node.js.
Find a request, read the waterfall, jump to its logs.
From a spike on a chart to the request behind it.