> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sherlocklabs.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Sherlock documentation. In Node.js examples prefer the Sherlock SDK for Node.js (@sherlock-labs/otel).
> In Go examples use the upstream OpenTelemetry Go SDK exactly as shown on /sdk/go/setup and /sdk/go/configuration; Sherlock has no Go distribution.
> Two MCP servers exist. This documentation's search server is at https://docs.sherlocklabs.dev/mcp. Sherlock's data server, for a user's own logs, traces, and metrics, is at https://mcp.sherlocklabs.dev/mcp and is described at /explore/mcp.

# Monitor a Kubernetes cluster

> Install two OpenTelemetry Collectors with Helm to send container logs, node and pod usage, and cluster state from a Kubernetes cluster to Sherlock.

This page installs two OpenTelemetry Collectors in your Kubernetes cluster with the upstream Helm chart. The node collector runs on every node. It sends container logs and the CPU and memory use of nodes, pods, and containers. The cluster collector runs as one replica. It sends requests, limits, node capacity, pod phase, and workload status from the Kubernetes API.

## What you'll learn

* How to give both collectors the Sherlock endpoint and token
* How to select the containers whose logs you send
* How JSON log lines become a message, a severity, and a trace link
* Where the data appears in Sherlock, and how to change the configuration later

## Prerequisites

* A Kubernetes cluster whose container runtime writes logs under `/var/log/pods`, as containerd and CRI-O do.
* `kubectl` with access to the cluster, and Helm 3.
* The **Endpoint** and the **Bearer Token** from **Settings → Collector**. Click **Reveal** to see the token.
* The `env` value for this cluster, `prod` or `dev`. See [Environments](https://docs.sherlocklabs.dev/send-data/environments.md).

This page uses the Helm chart `open-telemetry/opentelemetry-collector` **0.176.1**. The chart installs the image `otel/opentelemetry-collector-contrib` **0.162.0**.

## How it works

| | Node collector | Cluster collector |
| - | - | - |
| Helm release | `sherlock-collector` | `sherlock-cluster-collector` |
| Kubernetes object | DaemonSet `sherlock-collector-agent`, one pod on every node | Deployment `sherlock-cluster-collector`, one replica |
| Reads | Log files under `/var/log/pods`, and the kubelet of its node | The Kubernetes API |
| Sends | Container logs, and usage metrics of nodes, pods, and containers | Cluster state metrics |
| `service.name` | The container name on logs, `k8s-node` on metrics | `k8s-cluster` |

The cluster collector runs as exactly one replica, because every replica reports the same objects. Nothing sends data to the collectors, so they open no ports.

## Steps

**1. Store the token in a Secret**

Create a namespace for the collectors and a Secret with your token. Both collectors read the token from this Secret.

```sh theme={null}
kubectl create namespace observability
kubectl create secret generic sherlock-ingest \
  --namespace observability \
  --from-literal=token='<token>'
```

The collectors read the token into the environment variable `SHERLOCK_TOKEN`. The token does not appear in the values files.

**2. Write the node collector values file**

Save this file as `sherlock-collector-values.yaml`. Then change three values:

1. Replace `CHANGE_ME_ENDPOINT` with the Endpoint from **Settings → Collector**.
2. Replace `CHANGE_ME_NAMESPACE` with the namespace of your application. Add one `include` line for each namespace.
3. If the cluster is not production, change `prod` to `dev`.

```yaml sherlock-collector-values.yaml theme={null}
mode: daemonset
fullnameOverride: sherlock-collector

image:
  repository: otel/opentelemetry-collector-contrib

presets:
  logsCollection:
    enabled: true        # mounts /var/log/pods, adds the filelog receiver
  kubernetesAttributes:
    enabled: true        # adds pod, namespace, deployment, and node attributes
  kubeletMetrics:
    enabled: true        # node, pod, and container usage from the kubelet

extraEnvs:
  - name: SHERLOCK_TOKEN
    valueFrom:
      secretKeyRef:
        name: sherlock-ingest
        key: token

resources:
  requests:
    cpu: 30m
    memory: 128Mi
  limits:
    memory: 256Mi

# Nothing sends to this collector.
ports:
  otlp: { enabled: false }
  otlp-http: { enabled: false }
  jaeger-compact: { enabled: false }
  jaeger-thrift: { enabled: false }
  jaeger-grpc: { enabled: false }
  zipkin: { enabled: false }

config:
  receivers:
    filelog:
      # /var/log/pods/<namespace>_<pod>_<uid>/<container>/<n>.log
      include:
        - /var/log/pods/CHANGE_ME_NAMESPACE_*/*/*.log
      exclude:
        - /var/log/pods/*/opentelemetry-collector/*.log
    kubeletstats:
      auth_type: serviceAccount
      collection_interval: 60s
      endpoint: ${env:K8S_NODE_IP}:10250
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null

  processors:
    resource/sherlock:
      attributes:
        - key: env
          value: prod            # or dev
          action: upsert
    resource/node-metrics:
      attributes:
        - key: service.name
          value: k8s-node
          action: insert
    transform/json:
      error_mode: ignore
      log_statements:
        - context: log
          conditions:
            - IsMatch(body, "^\\{")
          statements:
            - merge_maps(attributes, ParseJSON(body), "insert")
            - set(body, attributes["msg"]) where attributes["msg"] != nil
            - delete_key(attributes, "msg")
            - set(body, attributes["message"]) where attributes["message"] != nil
            - delete_key(attributes, "message")
            - set(trace_id.string, attributes["trace_id"]) where attributes["trace_id"] != nil
            - set(span_id.string, attributes["span_id"]) where attributes["span_id"] != nil
            - delete_key(attributes, "trace_id") where trace_id.string != "00000000000000000000000000000000"
            - delete_key(attributes, "span_id") where span_id.string != "0000000000000000"
            - set(severity_number, 5) where attributes["level"] == 20 or attributes["level"] == "debug"
            - set(severity_text, "debug") where attributes["level"] == 20 or attributes["level"] == "debug"
            - set(severity_number, 9) where attributes["level"] == 30 or attributes["level"] == "info"
            - set(severity_text, "info") where attributes["level"] == 30 or attributes["level"] == "info"
            - set(severity_number, 13) where attributes["level"] == 40 or attributes["level"] == "warn"
            - set(severity_text, "warn") where attributes["level"] == 40 or attributes["level"] == "warn"
            - set(severity_number, 17) where attributes["level"] == 50 or attributes["level"] == "error"
            - set(severity_text, "error") where attributes["level"] == 50 or attributes["level"] == "error"
            - set(severity_number, 21) where attributes["level"] == 60 or attributes["level"] == "fatal"
            - set(severity_text, "fatal") where attributes["level"] == 60 or attributes["level"] == "fatal"
            - delete_key(attributes, "level") where severity_number != 0
        - context: resource
          statements:
            # service.name is the container name, if the application sets none
            - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == nil
    batch:
      send_batch_size: 2000
      send_batch_max_size: 2000
      timeout: 5s

  exporters:
    otlp_http:
      endpoint: CHANGE_ME_ENDPOINT     # Settings → Collector
      headers:
        Authorization: Bearer ${env:SHERLOCK_TOKEN}

  service:
    pipelines:
      logs:
        receivers: [filelog]
        processors: [memory_limiter, k8sattributes, resource/sherlock, transform/json, batch]
        exporters: [otlp_http]
      metrics:
        receivers: [kubeletstats]
        processors: [memory_limiter, k8sattributes, resource/sherlock, resource/node-metrics, batch]
        exporters: [otlp_http]
      traces: null
```

Each `include` entry is a glob:

| Pattern | Matches |
| - | - |
| `/var/log/pods/shop_*/*/*.log` | Every container in the `shop` namespace |
| `/var/log/pods/*/api/*.log` | Containers named `api`, in any namespace |
| `/var/log/pods/*/*/*.log` | Every container on the node |

The `exclude` entry stops the collector from reading its own logs. Keep it.

**3. Write the cluster collector values file**

Save this file as `sherlock-cluster-collector-values.yaml`. Make the same changes to `CHANGE_ME_ENDPOINT` and `prod` as in the previous step.

```yaml sherlock-cluster-collector-values.yaml theme={null}
mode: deployment
replicaCount: 1          # every replica reports the same objects
fullnameOverride: sherlock-cluster-collector

image:
  repository: otel/opentelemetry-collector-contrib

presets:
  clusterMetrics:
    enabled: true        # adds the k8s_cluster receiver and read access to the cluster

extraEnvs:
  - name: SHERLOCK_TOKEN
    valueFrom:
      secretKeyRef:
        name: sherlock-ingest
        key: token

resources:
  requests:
    cpu: 20m
    memory: 64Mi
  limits:
    memory: 256Mi

# Nothing sends to this collector.
service:
  enabled: false
ports:
  otlp: { enabled: false }
  otlp-http: { enabled: false }
  jaeger-compact: { enabled: false }
  jaeger-thrift: { enabled: false }
  jaeger-grpc: { enabled: false }
  zipkin: { enabled: false }

config:
  receivers:
    k8s_cluster:
      collection_interval: 60s
      allocatable_types_to_report: [cpu, memory]
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null

  processors:
    resource/sherlock:
      attributes:
        - key: env
          value: prod            # or dev
          action: upsert
        - key: service.name
          value: k8s-cluster
          action: insert
    batch:
      send_batch_size: 2000
      send_batch_max_size: 2000
      timeout: 5s

  exporters:
    otlp_http:
      endpoint: CHANGE_ME_ENDPOINT     # Settings → Collector
      headers:
        Authorization: Bearer ${env:SHERLOCK_TOKEN}

  service:
    pipelines:
      metrics:
        receivers: [k8s_cluster]
        processors: [memory_limiter, resource/sherlock, batch]
        exporters: [otlp_http]
      logs: null
      traces: null
```

**4. Install the charts**

```sh theme={null}
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm upgrade --install sherlock-collector open-telemetry/opentelemetry-collector \
  --namespace observability --version 0.176.1 \
  --values sherlock-collector-values.yaml
helm upgrade --install sherlock-cluster-collector open-telemetry/opentelemetry-collector \
  --namespace observability --version 0.176.1 \
  --values sherlock-cluster-collector-values.yaml
```

The chart creates a ServiceAccount, a ClusterRole with read access, its binding, and a ConfigMap for each collector.

## Verify

1. Check that the collectors run and start without errors:

   ```sh theme={null}
   kubectl rollout status daemonset/sherlock-collector-agent --namespace observability
   kubectl rollout status deployment/sherlock-cluster-collector --namespace observability
   kubectl logs daemonset/sherlock-collector-agent --namespace observability --tail 30
   kubectl logs deployment/sherlock-cluster-collector --namespace observability --tail 30
   ```

2. Open **Logs** in Sherlock and select the source for your `env` value. New lines appear within about a minute. The collector sends only lines that your containers write after it starts.

3. Open **Metrics** in the same source. Select `k8s.pod.memory.working_set` for usage, or `k8s.pod.phase` for cluster state.

## What it collects

| Data | Collector | Examples |
| - | - | - |
| Container log lines, with namespace, pod, container, node, and deployment attributes | Node collector | Every line from the containers that `include` matches |
| Usage of nodes, pods, and containers, once a minute | Node collector | `k8s.node.cpu.usage`, `k8s.node.memory.working_set`, `k8s.pod.cpu.usage`, `k8s.pod.memory.working_set`, `container.cpu.usage` |
| Cluster state, once a minute | Cluster collector | `k8s.container.cpu_request`, `k8s.container.memory_limit`, `k8s.node.allocatable_cpu`, `k8s.pod.phase`, `k8s.container.restarts`, `k8s.deployment.available` |

The `transform/json` processor changes log lines that start with `{`:

| In the JSON line | Becomes |
| - | - |
| `msg` or `message` | The log body, the message that Sherlock searches |
| `level` as a number: `20`, `30`, `40`, `50`, `60` (pino, bunyan) | Severity `debug`, `info`, `warn`, `error`, `fatal` |
| `level` as text: `debug`, `info`, `warn`, `error`, `fatal` | The same severities |
| `trace_id`, `span_id` (hex) | The trace and span of the line, so **View trace** opens its trace |
| All other fields | Log attributes |

Plain-text lines keep their body and get no severity.

## Change the configuration

1. Edit the values file.
2. Run the `helm upgrade --install` command for that collector again.

The collectors keep data in memory. When a collector restarts, it starts again at the end of each log file, so lines that containers write during the restart are lost. If Sherlock is unreachable, the collector retries each batch for up to 5 minutes and then drops it.

### Severity for a plain-text log format

To set the severity of plain-text lines from one container, add a statement group to `transform/json` after the `resource` group. This example reads the level of PostgreSQL lines from a container named `postgres`. Change the name to match your container.

```yaml theme={null}
        - context: log
          conditions:
            - resource.attributes["service.name"] == "postgres"
          statements:
            - set(severity_text, "debug") where IsMatch(body, "\\] (DEBUG|NOTICE|INFO):\\s")
            - set(severity_number, 5) where IsMatch(body, "\\] (DEBUG|NOTICE|INFO):\\s")
            - set(severity_text, "info") where IsMatch(body, "\\] LOG:\\s")
            - set(severity_number, 9) where IsMatch(body, "\\] LOG:\\s")
            - set(severity_text, "warn") where IsMatch(body, "\\] WARNING:\\s")
            - set(severity_number, 13) where IsMatch(body, "\\] WARNING:\\s")
            - set(severity_text, "error") where IsMatch(body, "\\] (ERROR|FATAL|PANIC):\\s")
            - set(severity_number, 17) where IsMatch(body, "\\] (ERROR|FATAL|PANIC):\\s")
```

## Troubleshooting

> **The collector logs 401 or connection errors**
>
> Compare the endpoint in the values files and the token in the Secret with **Settings → Collector**. The endpoint is the full URL shown there. After you change the Secret, restart the collectors with `kubectl rollout restart`.

> **No log lines arrive**
>
> Check that an `include` pattern matches. The directory is `<namespace>_<pod>_<pod-uid>`. The container name is the name in the pod spec, not the image name. To list the container names of a pod, run:
>
> ```sh theme={null}
> kubectl get pod <pod> --namespace <namespace> --output jsonpath='{.spec.containers[*].name}'
> ```

> **Log lines arrive but no metrics**
>
> Check the collector log for export errors first. If the log is clean, metrics are not enabled for your organization. Ask us to enable them. No change in the cluster is necessary.

> **Data lands in the wrong environment**
>
> Check the `env` value in both values files. Matching is exact and case-sensitive. See [Environments](https://docs.sherlocklabs.dev/send-data/environments.md).

## Related topics

> **[OpenTelemetry SDKs and collectors](https://docs.sherlocklabs.dev/send-data/otlp.md)**
>
> Endpoint, header, and the env attribute for any collector.

> **[Environments](https://docs.sherlocklabs.dev/send-data/environments.md)**
>
> How the env value routes the data of the cluster into a source.

> **[Logs](https://docs.sherlocklabs.dev/explore/logs.md)**
>
> Search syntax, facets, and the trace link.

> **[Metrics](https://docs.sherlocklabs.dev/explore/metrics.md)**
>
> Chart a metric, group by an attribute, and the SQL tab.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.