Observability: Logs, Traces, And Soul Timelines

A web service that's slow is obvious — pages don't load. A soul that's quietly drifting — choosing the wrong skill, looping on the same heartbeat, burning model budget while you sleep — is invisible until you check. OpenClaw is opinionated here: every heartbeat emits a structured log, every skill call emits a trace span, and every soul has a timeline view in Mission Control. Use them or you're flying blind.

Three layers, three questions

What to surface in logs

OpenClaw's structured logs include heartbeat ID, soul slug, model used, token count, skill calls, duration, and outcome. JSON-shaped, one line per event. The default level is info — keep it there. Cranking to debug spams useful patterns into noise; bumping to warn hides exactly the boring-success events you need to spot the abnormal one.

{ "ts": "2026-04-27T08:00:00.123Z", "level": "info", "event": "heartbeat.complete", "soul": "inbox-triage", "heartbeat_id": "hb_2k4n9", "interval_s": 900, "actual_duration_s": 12.4, "model": "qwen3.5:8b", "tokens_in": 4218, "tokens_out": 612, "skills_called": ["gmail.list", "gmail.label"], "approvals_pending": 0, "outcome": "success" }

Traces: where the time actually went

A heartbeat looks like a single event in logs but is a tree of work — model call, skill call, sub-skill call, return. OTLP traces give you that tree. OpenClaw exports OpenTelemetry by default; point it at a collector (Jaeger locally, Honeycomb or Vercel Observability for hosted) and you get flame graphs of every heartbeat. The first time a soul feels 'slow,' a trace shows you it's the model — or it's that one skill that's quietly making three round-trips. Don't guess.

The soul timeline

Mission Control's soul-timeline view is the long-running version of the trace. It plots heartbeats over hours and days — interval, duration, outcome, token spend. Patterns you can only see here: a soul whose duration is creeping up day over day (memory bloat), a soul whose token-per-tick has 10x'd since you swapped models, a soul whose interval drifts because heartbeats run longer than the gap between them.

Sketch your dashboard before you build it

1. 1Top row: number of healthy souls, number with pending approvals, number with errors in last hour. Big numbers, no charts.
2. 2Per-soul row: last heartbeat timestamp, last duration, last token cost, status dot (green / yellow / red).
3. 3Trend chart: tokens-per-day for each soul, last 7 days. Spot a soul whose model swap doubled its cost overnight.
4. 4Heartbeat-anomaly chart: actual_duration vs interval, log scale. Anything trending toward 1.0 is a soul that's about to overlap itself.
5. 5Audit feed: scrolling list of skill calls, last 50. The chaos-monkey check — does what's happening match what you authorized?

Alerting on heartbeat anomalies

The high-leverage alerts are not 'soul errored' — those are loud and self-announcing. The ones that catch real problems are the silent failures: a soul that hasn't ticked, a soul whose tick is taking longer than its interval, a soul whose token spend doubled overnight without a model change. Wire these as paging-grade alerts; everything else is dashboard-grade.

Apply: instrument one soul this week

1. 1Pick the soul that runs most often.
2. 2Tail its logs for one full heartbeat — read every field. If anything's missing that you'd want at 3am, raise a feature request or add a log line.
3. 3Wire OTLP to a free Honeycomb / Jaeger / SigNoz collector. Look at one trace.
4. 4Sketch your one-screen dashboard on paper before you open Grafana.
5. 5Set the 'heartbeat missed' and 'tick > interval' alerts. Skip the rest until you've used the dashboard for a week.

The big idea: a long-running agent without observability is a long-running mystery. Wire logs, traces, and the soul timeline before you trust a soul with anything that matters.