Monitor a Kubernetes cluster

This page installs two OpenTelemetry Collectors in your Kubernetes cluster with the upstream Helm chart. The node collector runs on every node. It sends container logs and the CPU and memory use of nodes, pods, and containers. The cluster collector runs as one replica. It sends requests, limits, node capacity, pod phase, and workload status from the Kubernetes API.

What you’ll learn

How to give both collectors the Sherlock endpoint and token
How to select the containers whose logs you send
How JSON log lines become a message, a severity, and a trace link
Where the data appears in Sherlock, and how to change the configuration later

Prerequisites

A Kubernetes cluster whose container runtime writes logs under /var/log/pods, as containerd and CRI-O do.
kubectl with access to the cluster, and Helm 3.
The Endpoint and the Bearer Token from Settings → Collector. Click Reveal to see the token.
The env value for this cluster, prod or dev. See Environments.

This page uses the Helm chart open-telemetry/opentelemetry-collector 0.176.1. The chart installs the image otel/opentelemetry-collector-contrib 0.162.0.

How it works

	Node collector	Cluster collector
Helm release	`sherlock-collector`	`sherlock-cluster-collector`
Kubernetes object	DaemonSet `sherlock-collector-agent`, one pod on every node	Deployment `sherlock-cluster-collector`, one replica
Reads	Log files under `/var/log/pods`, and the kubelet of its node	The Kubernetes API
Sends	Container logs, and usage metrics of nodes, pods, and containers	Cluster state metrics
`service.name`	The container name on logs, `k8s-node` on metrics	`k8s-cluster`

The cluster collector runs as exactly one replica, because every replica reports the same objects. Nothing sends data to the collectors, so they open no ports.

Steps

Store the token in a Secret

Create a namespace for the collectors and a Secret with your token. Both collectors read the token from this Secret.

kubectl create namespace observability
kubectl create secret generic sherlock-ingest \
  --namespace observability \
  --from-literal=token='<token>'

The collectors read the token into the environment variable SHERLOCK_TOKEN. The token does not appear in the values files.

Write the node collector values file

Save this file as sherlock-collector-values.yaml. Then change three values:

Replace CHANGE_ME_ENDPOINT with the Endpoint from Settings → Collector.
Replace CHANGE_ME_NAMESPACE with the namespace of your application. Add one include line for each namespace.
If the cluster is not production, change prod to dev.

sherlock-collector-values.yaml

mode: daemonset
fullnameOverride: sherlock-collector

image:
  repository: otel/opentelemetry-collector-contrib

presets:
  logsCollection:
    enabled: true        # mounts /var/log/pods, adds the filelog receiver
  kubernetesAttributes:
    enabled: true        # adds pod, namespace, deployment, and node attributes
  kubeletMetrics:
    enabled: true        # node, pod, and container usage from the kubelet

extraEnvs:
  - name: SHERLOCK_TOKEN
    valueFrom:
      secretKeyRef:
        name: sherlock-ingest
        key: token

resources:
  requests:
    cpu: 30m
    memory: 128Mi
  limits:
    memory: 256Mi

# Nothing sends to this collector.
ports:
  otlp: { enabled: false }
  otlp-http: { enabled: false }
  jaeger-compact: { enabled: false }
  jaeger-thrift: { enabled: false }
  jaeger-grpc: { enabled: false }
  zipkin: { enabled: false }

config:
  receivers:
    filelog:
      # /var/log/pods/<namespace>_<pod>_<uid>/<container>/<n>.log
      include:
        - /var/log/pods/CHANGE_ME_NAMESPACE_*/*/*.log
      exclude:
        - /var/log/pods/*/opentelemetry-collector/*.log
    kubeletstats:
      auth_type: serviceAccount
      collection_interval: 60s
      endpoint: ${env:K8S_NODE_IP}:10250
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null

  processors:
    resource/sherlock:
      attributes:
        - key: env
          value: prod            # or dev
          action: upsert
    resource/node-metrics:
      attributes:
        - key: service.name
          value: k8s-node
          action: insert
    transform/json:
      error_mode: ignore
      log_statements:
        - context: log
          conditions:
            - IsMatch(body, "^\\{")
          statements:
            - merge_maps(attributes, ParseJSON(body), "insert")
            - set(body, attributes["msg"]) where attributes["msg"] != nil
            - delete_key(attributes, "msg")
            - set(body, attributes["message"]) where attributes["message"] != nil
            - delete_key(attributes, "message")
            - set(trace_id.string, attributes["trace_id"]) where attributes["trace_id"] != nil
            - set(span_id.string, attributes["span_id"]) where attributes["span_id"] != nil
            - delete_key(attributes, "trace_id") where trace_id.string != "00000000000000000000000000000000"
            - delete_key(attributes, "span_id") where span_id.string != "0000000000000000"
            - set(severity_number, 5) where attributes["level"] == 20 or attributes["level"] == "debug"
            - set(severity_text, "debug") where attributes["level"] == 20 or attributes["level"] == "debug"
            - set(severity_number, 9) where attributes["level"] == 30 or attributes["level"] == "info"
            - set(severity_text, "info") where attributes["level"] == 30 or attributes["level"] == "info"
            - set(severity_number, 13) where attributes["level"] == 40 or attributes["level"] == "warn"
            - set(severity_text, "warn") where attributes["level"] == 40 or attributes["level"] == "warn"
            - set(severity_number, 17) where attributes["level"] == 50 or attributes["level"] == "error"
            - set(severity_text, "error") where attributes["level"] == 50 or attributes["level"] == "error"
            - set(severity_number, 21) where attributes["level"] == 60 or attributes["level"] == "fatal"
            - set(severity_text, "fatal") where attributes["level"] == 60 or attributes["level"] == "fatal"
            - delete_key(attributes, "level") where severity_number != 0
        - context: resource
          statements:
            # service.name is the container name, if the application sets none
            - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == nil
    batch:
      send_batch_size: 2000
      send_batch_max_size: 2000
      timeout: 5s

  exporters:
    otlp_http:
      endpoint: CHANGE_ME_ENDPOINT     # Settings → Collector
      headers:
        Authorization: Bearer ${env:SHERLOCK_TOKEN}

  service:
    pipelines:
      logs:
        receivers: [filelog]
        processors: [memory_limiter, k8sattributes, resource/sherlock, transform/json, batch]
        exporters: [otlp_http]
      metrics:
        receivers: [kubeletstats]
        processors: [memory_limiter, k8sattributes, resource/sherlock, resource/node-metrics, batch]
        exporters: [otlp_http]
      traces: null

Each include entry is a glob:

Pattern	Matches
`/var/log/pods/shop_//*.log`	Every container in the `shop` namespace
`/var/log/pods//api/.log`	Containers named `api`, in any namespace
`/var/log/pods///*.log`	Every container on the node

The exclude entry stops the collector from reading its own logs. Keep it.

Write the cluster collector values file

Save this file as sherlock-cluster-collector-values.yaml. Make the same changes to CHANGE_ME_ENDPOINT and prod as in the previous step.

sherlock-cluster-collector-values.yaml

mode: deployment
replicaCount: 1          # every replica reports the same objects
fullnameOverride: sherlock-cluster-collector

image:
  repository: otel/opentelemetry-collector-contrib

presets:
  clusterMetrics:
    enabled: true        # adds the k8s_cluster receiver and read access to the cluster

extraEnvs:
  - name: SHERLOCK_TOKEN
    valueFrom:
      secretKeyRef:
        name: sherlock-ingest
        key: token

resources:
  requests:
    cpu: 20m
    memory: 64Mi
  limits:
    memory: 256Mi

# Nothing sends to this collector.
service:
  enabled: false
ports:
  otlp: { enabled: false }
  otlp-http: { enabled: false }
  jaeger-compact: { enabled: false }
  jaeger-thrift: { enabled: false }
  jaeger-grpc: { enabled: false }
  zipkin: { enabled: false }

config:
  receivers:
    k8s_cluster:
      collection_interval: 60s
      allocatable_types_to_report: [cpu, memory]
    otlp: null
    jaeger: null
    zipkin: null
    prometheus: null

  processors:
    resource/sherlock:
      attributes:
        - key: env
          value: prod            # or dev
          action: upsert
        - key: service.name
          value: k8s-cluster
          action: insert
    batch:
      send_batch_size: 2000
      send_batch_max_size: 2000
      timeout: 5s

  exporters:
    otlp_http:
      endpoint: CHANGE_ME_ENDPOINT     # Settings → Collector
      headers:
        Authorization: Bearer ${env:SHERLOCK_TOKEN}

  service:
    pipelines:
      metrics:
        receivers: [k8s_cluster]
        processors: [memory_limiter, resource/sherlock, batch]
        exporters: [otlp_http]
      logs: null
      traces: null

Install the charts

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm upgrade --install sherlock-collector open-telemetry/opentelemetry-collector \
  --namespace observability --version 0.176.1 \
  --values sherlock-collector-values.yaml
helm upgrade --install sherlock-cluster-collector open-telemetry/opentelemetry-collector \
  --namespace observability --version 0.176.1 \
  --values sherlock-cluster-collector-values.yaml

The chart creates a ServiceAccount, a ClusterRole with read access, its binding, and a ConfigMap for each collector.

Verify

Check that the collectors run and start without errors:

kubectl rollout status daemonset/sherlock-collector-agent --namespace observability
kubectl rollout status deployment/sherlock-cluster-collector --namespace observability
kubectl logs daemonset/sherlock-collector-agent --namespace observability --tail 30
kubectl logs deployment/sherlock-cluster-collector --namespace observability --tail 30

Open Logs in Sherlock and select the source for your env value. New lines appear within about a minute. The collector sends only lines that your containers write after it starts.
Open Metrics in the same source. Select k8s.pod.memory.working_set for usage, or k8s.pod.phase for cluster state.

What it collects

Data	Collector	Examples
Container log lines, with namespace, pod, container, node, and deployment attributes	Node collector	Every line from the containers that `include` matches
Usage of nodes, pods, and containers, once a minute	Node collector	`k8s.node.cpu.usage`, `k8s.node.memory.working_set`, `k8s.pod.cpu.usage`, `k8s.pod.memory.working_set`, `container.cpu.usage`
Cluster state, once a minute	Cluster collector	`k8s.container.cpu_request`, `k8s.container.memory_limit`, `k8s.node.allocatable_cpu`, `k8s.pod.phase`, `k8s.container.restarts`, `k8s.deployment.available`

The transform/json processor changes log lines that start with {:

In the JSON line	Becomes
`msg` or `message`	The log body, the message that Sherlock searches
`level` as a number: `20`, `30`, `40`, `50`, `60` (pino, bunyan)	Severity `debug`, `info`, `warn`, `error`, `fatal`
`level` as text: `debug`, `info`, `warn`, `error`, `fatal`	The same severities
`trace_id`, `span_id` (hex)	The trace and span of the line, so View trace opens its trace
All other fields	Log attributes

Plain-text lines keep their body and get no severity.

Change the configuration

Edit the values file.
Run the helm upgrade --install command for that collector again.

The collectors keep data in memory. When a collector restarts, it starts again at the end of each log file, so lines that containers write during the restart are lost. If Sherlock is unreachable, the collector retries each batch for up to 5 minutes and then drops it.

Severity for a plain-text log format

To set the severity of plain-text lines from one container, add a statement group to transform/json after the resource group. This example reads the level of PostgreSQL lines from a container named postgres. Change the name to match your container.

        - context: log
          conditions:
            - resource.attributes["service.name"] == "postgres"
          statements:
            - set(severity_text, "debug") where IsMatch(body, "\\] (DEBUG|NOTICE|INFO):\\s")
            - set(severity_number, 5) where IsMatch(body, "\\] (DEBUG|NOTICE|INFO):\\s")
            - set(severity_text, "info") where IsMatch(body, "\\] LOG:\\s")
            - set(severity_number, 9) where IsMatch(body, "\\] LOG:\\s")
            - set(severity_text, "warn") where IsMatch(body, "\\] WARNING:\\s")
            - set(severity_number, 13) where IsMatch(body, "\\] WARNING:\\s")
            - set(severity_text, "error") where IsMatch(body, "\\] (ERROR|FATAL|PANIC):\\s")
            - set(severity_number, 17) where IsMatch(body, "\\] (ERROR|FATAL|PANIC):\\s")

Troubleshooting

The collector logs 401 or connection errors

Compare the endpoint in the values files and the token in the Secret with Settings → Collector. The endpoint is the full URL shown there. After you change the Secret, restart the collectors with kubectl rollout restart.

No log lines arrive

Check that an include pattern matches. The directory is <namespace>_<pod>_<pod-uid>. The container name is the name in the pod spec, not the image name. To list the container names of a pod, run:

kubectl get pod <pod> --namespace <namespace> --output jsonpath='{.spec.containers[*].name}'

Log lines arrive but no metrics

Check the collector log for export errors first. If the log is clean, metrics are not enabled for your organization. Ask us to enable them. No change in the cluster is necessary.

Data lands in the wrong environment

Check the env value in both values files. Matching is exact and case-sensitive. See Environments.

OpenTelemetry SDKs and collectors

Endpoint, header, and the env attribute for any collector.

Environments

How the env value routes the data of the cluster into a source.

Logs

Search syntax, facets, and the trace link.

Metrics

Chart a metric, group by an attribute, and the SQL tab.

Send data

Node.js SDK

Go SDK

Monitor a Kubernetes cluster

What you’ll learn

Prerequisites

How it works

Steps

Verify

What it collects

Change the configuration

Severity for a plain-text log format

Troubleshooting

OpenTelemetry SDKs and collectors

Environments

Logs

Metrics

​What you’ll learn

​Prerequisites

​How it works

​Steps

​Verify

​What it collects

​Change the configuration

​Severity for a plain-text log format

​Troubleshooting

​Related topics

OpenTelemetry SDKs and collectors

Environments

Logs

Metrics

What you’ll learn

Prerequisites

How it works

Steps

Verify

What it collects

Change the configuration

Severity for a plain-text log format

Troubleshooting

Related topics