Trace

Overview

The trace plugin gives you visibility into what anna is doing under the hood. Every LLM call, tool execution, and memory operation is tracked with timing, token usage, and error details.

It operates in two modes:

• Log mode (always on) -- structured log lines via Go's slog, visible in stderr. Zero configuration needed.
• OpenTelemetry mode (opt-in) -- exports distributed traces via standard OTLP environment variables. Both OTLP/gRPC and OTLP/HTTP are supported, including authenticated backends. Activate by setting an OTLP endpoint.

Configuration

Log Mode

Log mode is always active when the plugin is enabled. Control verbosity with LOG_LEVEL:

# Default -- LLM/tool/memory events at INFO
anna serve

# Verbose -- includes memory detail fields
LOG_LEVEL=TRACE anna serve

Example log output:

level=INFO msg=post_llm_call hook=trace provider=anthropic model=claude-sonnet-4-20250514 stop_reason=tool_use duration=3.2s ttft=450ms input_tokens=12500 output_tokens=350 cache_read=8000
level=INFO msg=post_tool_call hook=trace tool=bash call_id=call_01 is_error=false duration=1.5s result_len=256
level=INFO msg=post_memory_call hook=trace op=compact duration=200ms token_count=8000 token_delta=-4500

OpenTelemetry Mode

Set OTEL_EXPORTER_OTLP_ENDPOINT to enable. Anna delegates exporter configuration to the OpenTelemetry SDK, so standard OTel environment variables are supported:

When OTel is enabled, both modes run simultaneously -- you get log lines and exported traces.

Common Pitfalls

• Always include a scheme in OTEL_EXPORTER_OTLP_ENDPOINT. Use https://collector.example.com:4317, not collector.example.com:4317. Omitting the scheme can produce malformed export URLs such as http:///v1/traces.
• Use the correct protocol for your backend. OTLP/HTTP usually needs OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; OTLP/gRPC usually uses OTEL_EXPORTER_OTLP_PROTOCOL=grpc.
• For OTLP/HTTP, set the base path, not /v1/traces. The exporter appends /v1/traces automatically.
• Do not set OTEL_EXPORTER_OTLP_INSECURE=true for TLS endpoints. Secure collectors should use OTEL_EXPORTER_OTLP_INSECURE=false.
• Header values are comma-separated key=value pairs without shell quotes inside the value. Example: authorization=Basic abc123,organization=default.

Using with Jaeger

Jaeger is the simplest way to visualize anna's traces. It accepts OTLP natively and runs as a single container.

# Start Jaeger
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/jaeger:latest

# Start anna with tracing over OTLP/gRPC
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
anna serve

Open http://localhost:16686, select the anna service, and click Find Traces. Each chat session appears as a trace with a waterfall view of LLM calls, tool executions, and memory operations.

Using with Other Backends

Any OTLP-compatible backend works. Examples:

# Grafana Tempo over OTLP/gRPC
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.internal:4317 \
anna serve

# SigNoz over OTLP/gRPC
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz.internal:4317 \
anna serve

# OTel Collector over OTLP/gRPC
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
anna serve

# Cloud OTLP/gRPC with TLS and auth header
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vendor.com:443 \
OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=Bearer <token>" \
OTEL_EXPORTER_OTLP_INSECURE=false \
anna serve

# Cloud OTLP/HTTP with TLS and auth header
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.vendor.com/api/default \
OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=Basic <base64>,organization=default,stream-name=default" \
OTEL_EXPORTER_OTLP_INSECURE=false \
anna serve

If your provider gives you an OTLP/HTTP endpoint such as https://collector.example.com/api/default, use that exact base URL and let the exporter append /v1/traces.

What Gets Traced

LLM Calls

Every call to an LLM provider is captured as a gen_ai.chat span:

• Model name (requested and actual)
• Provider (anthropic, openai, etc.)
• Token usage: input, output, cache read, cache write
• Time to first token (TTFT)
• Total duration
• Stop reason (end_turn, tool_use, max_tokens, etc.)
• Errors

Tool Executions

Each tool call is captured as a gen_ai.execute_tool span:

• Tool name (bash, read, write, edit, webfetch, agent, etc.)
• Call ID
• Duration
• Success or failure

Memory Operations

Memory operations (append, assemble, compact, search, etc.) are captured as memory.* spans:

• Operation type
• Duration
• Token and message counts
• Token delta (for compaction)
• Errors

Sandbox Lifecycle

Sandbox startup is captured with sandbox.* spans so boxsh failures can be traced past the final broken-pipe symptom:

• Session creation in the runner
• Backend startup and overlay/session directory setup
• Boxsh client process startup
• JSON-RPC handshake
• Session close or liveness loss reason

These spans include Anna-specific attributes such as sandbox backend, source/destination roots, working directory, network mode, read-only bind count, and captured error type.

Trace Structure

Spans are organized into a hierarchy per chat session:

chat
  └── turn 1
       ├── gen_ai.chat                 3.2s
       ├── gen_ai.execute_tool (bash)  1.5s
       ├── gen_ai.execute_tool (read)  0.1s
       └── memory.append               0.02s
  └── turn 2
       ├── gen_ai.chat                 2.8s
       └── memory.compact              0.2s

A new turn starts each time anna calls the LLM. The chat root span covers the entire conversation and closes after 2 minutes of inactivity.

Span Attributes Reference

LLM and tool spans follow OpenTelemetry GenAI semantic conventions:

Memory spans use anna-specific attributes:

Sandbox lifecycle spans use these Anna-specific attributes:

Managing the Plugin

The trace plugin is enabled by default.

anna plugin list                   # Check status
anna plugin disable hook/trace     # Disable all tracing
anna plugin enable hook/trace      # Re-enable

Disabling the trace plugin turns off both log mode and OTel mode. LLM calls, tool executions, and memory operations will no longer be logged or exported.