Observability

This guide walks through tracing your AI agent in production with the Ashr Labs SDK. It covers everything from getting your first trace into the dashboard, to instrumenting tool calls and LLM generations, to reading analytics back out, to wiring up a realtime voice agent on LiveKit.

This is a separate product from the Testing Platform. The testing platform (datasets, eval runs, RunBuilder, EvalRunner) is for offline evaluation. Observability is for tracing your agent in production. They share the same SDK and one API key but are independent features. Requires the observability feature flag to be enabled for your tenant.

Overview

Production observability has three building blocks:

1. Trace — one user-facing interaction (a chat turn, a job run, a phone call). Top-level container.
2. Span / Generation — one unit of work inside a trace (a tool invocation, a retrieval step, an LLM call). Spans nest arbitrarily; Generation is a Span subclass that also captures token usage and model.
3. Event — a point-in-time record (a guardrail check, a feature flag, a cache hit). No duration.

┌────────────────────────────────────────────────────────────┐
│ Trace: "handle-support-ticket"                             │
│ ┌──────────────────┐  ┌──────────────────┐  ┌───────────┐ │
│ │ Generation:      │  │ Span:            │  │ Event:    │ │
│ │ classify-intent  │  │ tool:lookup_acct │  │ guardrail │ │
│ └──────────────────┘  └──────────────────┘  └───────────┘ │
└────────────────────────────────────────────────────────────┘

Where it lands: traces flush to the Ashr Labs backend, get stored in Postgres, and render in the Observability panel of the Ashr Labs dashboard.

Production-safe by design. Tracing never raises into your code and never blocks your hot path — trace.end() enqueues the trace on a background flush and returns immediately. If the backend is unreachable, the failure is logged and surfaced via await trace.flush() (which resolves to an error object), never raised. Spans that are never closed are flushed at process exit.

The 3-Line Version

If you already have an API key, this is a complete instrumented agent turn:

import { AshrLabsClient } from "ashr-labs";

const client = new AshrLabsClient("tp_your_key_here");

await client.trace("handle-ticket", { userId: "user_42" }).wrap(async (trace) => {
  await trace.generation("answer", { model: "claude-sonnet-4-6" }).wrap(async (gen) => {
    const reply = await callLLM(/* ... */);
    gen.end({ output: reply, usage: { input_tokens: 50, output_tokens: 80 } });
  });
});

That's it. The trace flushes on exit, lands in your dashboard within a few seconds, and includes every span, generation, and event it contains. The rest of this guide explains how to customize each piece.

---

Step 1: Initialize the Client

Same client as the testing platform — one API key works for both products.

import { AshrLabsClient } from "ashr-labs";

const client = new AshrLabsClient("tp_your_api_key_here");

// Or load from environment
// const client = AshrLabsClient.fromEnv(); // reads ASHR_LABS_API_KEY

If you don't have a key yet, see the Quick Start for how to mint one. The first client.trace(...) call lazily resolves your tenant from the API key — no extra setup.

---

Step 2: Open a Trace

Wrap each user-facing interaction in a Trace. The recommended pattern is .wrap() — trace.end() is called automatically when the callback resolves, even if your code throws.

await client
  .trace("handle-ticket", {
    userId: "user_42",                          // optional — for grouping by end-user
    sessionId: "conv_abc",                       // optional — for multi-turn conversations
    metadata: { version: "v3", channel: "web" },
    tags: ["prod", "premium-tier"],
  })
  .wrap(async (trace) => {
    // ...
  });

Parameters:

Parameter	Type	Required	Description
name	string	Yes	Logical name for this interaction (e.g. "handle-ticket", "summarize-document")
userId	string	No	End-user ID for grouping in the dashboard
sessionId	string	No	Conversation/session ID for multi-turn flows
metadata	object	No	Arbitrary JSON-serializable metadata
tags	string[]	No	Tags for filtering in the dashboard

trace.end() is non-blocking — it enqueues a background flush and returns immediately, so it never adds latency to your agent's hot path. The server-assigned traceId lands on trace.traceId once the flush completes; call await trace.flush() to block until then (rarely needed — usually you just fire and forget).

---

Step 3: Instrument LLM Calls (Generations)

Wrap every LLM call in a trace.generation(...). This captures the model, the prompt, the completion, and token usage.

await trace
  .generation("classify-intent", {
    model: "claude-sonnet-4-6",
    input: [{ role: "user", content: "I can't log in" }],
    metadata: { temperature: 0.3 },
  })
  .wrap(async (gen) => {
    const response = await anthropicClient.messages.create(/* ... */);
    gen.end({
      output: { role: "assistant", content: response.content[0].text },
      usage: {
        input_tokens: response.usage.input_tokens,
        output_tokens: response.usage.output_tokens,
      },
    });
  });

The model argument is what shows up in the Model Usage rollup on the dashboard. Use the canonical provider/model name (claude-sonnet-4-6, gpt-4.1-mini, gemini-2.0-flash, etc.).

---

Step 4: Instrument Tool Calls (Spans)

For everything else — tool invocations, retrieval steps, RAG lookups, guardrail checks — use trace.span(...).

await trace
  .span("tool:lookup_account", {
    input: { userId: "user_42" },
    metadata: { backend: "postgres" },
  })
  .wrap(async (tool) => {
    const result = await lookupAccount("user_42");
    tool.end({ output: { status: "active", tier: "premium" } });
  });

If the body throws, the span auto-ends with level: "ERROR" and the exception message is captured in statusMessage. The exception is re-thrown — tracing never swallows errors:

await trace.span("tool:external_api").wrap(async (tool) => {
  const response = await callExternalApi(/* ... */); // if this throws...
  tool.end({ output: response });
});
// ...the span ended with level: "ERROR" and the exception propagates here

---

Step 5: Nest Spans Arbitrarily

Spans can contain other spans and generations. Nesting maps the agent's actual call tree, which is what makes the timeline view useful.

await client.trace("handle-ticket").wrap(async (trace) => {
  await trace.span("retrieval").wrap(async (retrieval) => {
    await retrieval.span("vector-search").wrap(async (v) => {
      v.end({ output: { hits: 8 } });
    });
    await retrieval.generation("rerank", { model: "cohere-rerank-3" }).wrap(async (r) => {
      r.end({ output: { top_3: [] }, usage: { input_tokens: 200, output_tokens: 30 } });
    });
  });

  await trace.generation("compose-reply", { model: "claude-sonnet-4-6" }).wrap(async (gen) => {
    gen.end({ output: { text: "..." }, usage: { input_tokens: 1200, output_tokens: 180 } });
  });
});

The dashboard renders this as a tree, with each level's latency contribution visible at a glance.

---

Step 6: Record Events

For point-in-time facts (no duration), use trace.event(...) or span.event(...):

trace.event("guardrail:toxicity", { input: { toxic: false }, level: "DEFAULT" });
trace.event("cache:hit",          { input: { key: "user_42:profile" } });
trace.event("flag:new-prompt",    { input: { variant: "B" } });

Level	Meaning
"DEBUG"	Verbose; off by default in dashboards
"DEFAULT"	Normal informational events
"WARNING"	Suspicious but recovered
"ERROR"	Failed with consequences

Events show up inline in the timeline view alongside the spans and generations from the same trace.

---

Manual Instrumentation (No wrap())

If you can't use .wrap() callbacks (async generators, streaming responses, callbacks), call .end() yourself:

const trace = client.trace("support-chat", { userId: "user_42", sessionId: "conv_abc" });

const gen = trace.generation("classify-intent", {
  model: "claude-sonnet-4-6",
  input: [{ role: "user", content: "Reset my password" }],
});
gen.end({
  output: { intent: "password_reset" },
  usage: { input_tokens: 50, output_tokens: 12 },
});

const tool = trace.span("tool:reset_password", { input: { userId: "user_42" } });
tool.end({ output: { success: true } });

trace.event("guardrail-check", { input: { passed: true } });

trace.end({ output: { resolution: "password_reset_complete" } }); // enqueues, returns immediately
await trace.flush();                                              // block until the backend accepts it
console.log(trace.traceId);                                       // server-assigned ID

If a span goes out of scope without .end() being called, it's auto-ended when the trace is flushed (or at process exit) with whatever data was attached. The dashboard marks these as level: "DEFAULT" with a statusMessage: "[no-end]" indicator so you know they were dangling.

---

Reading Your Data

The same client exposes read-side APIs for querying the dashboard programmatically.

List recent traces

const result = await client.listObservabilityTraces({
  userId: "user_42",       // optional filter
  sessionId: "conv_abc",   // optional filter
  limit: 50,
  page: 1,
});
for (const t of result.traces as Record<string, unknown>[]) {
  console.log(t.name, t.trace_id, t.start_time);
}

Get a single trace with its full tree

const detail = await client.getObservabilityTrace("abc123...");
const trace = detail.trace as Record<string, unknown>;
for (const obs of trace.observations as Record<string, unknown>[]) {
  console.log(obs.name, obs.type, obs.model, obs.usage);
}

Aggregate analytics

const analytics = await client.getObservabilityAnalytics(7);
const overview = analytics.overview as Record<string, number>;
console.log(`Traces: ${overview.total_traces}`);
console.log(`Tokens: ${overview.total_input_tokens} in / ${overview.total_output_tokens} out`);
console.log(`P95 latency: ${overview.p95_latency_ms}ms`);
console.log(`Error rate: ${overview.error_rate}`);

The overview object includes: total_traces, avg_latency_ms, p95_latency_ms, total_input_tokens, total_output_tokens, error_rate, total_tool_calls, unique_users, unique_sessions.

Errors and tool failures

const errors = await client.getObservabilityErrors({ days: 7, limit: 50 });
const toolErrors = await client.getObservabilityToolErrors({ days: 7, limit: 50 });

Both return { traces: [...], total: number } with the most recent failures first.

---

Voice Agents (Realtime / LiveKit)

For voice agents on LiveKit, the SDK ships a dedicated ashr_labs.voice_obs submodule that captures STT/LLM/TTS metrics, turn boundaries, barge-ins, mixed-audio replay, and per-stage cost from the AgentSession automatically.

Install with the LiveKit extra:

pip install ashr-labs[livekit]

Two-line attach in your worker:

import os
from ashr_labs.voice_obs.livekit import VoiceObservability

obs = VoiceObservability(api_key=os.environ["ASHR_LABS_API_KEY"])
obs.attach(session, agent_id="support_v3", agent_version="v42")

That's the entire instrumentation. STT, LLM, TTS metrics, turn boundaries, and barge-ins are all captured automatically by hooking the AgentSession's event surface. Mixed-audio replay is enabled by default — agent TTS and remote participant audio are mixed at 24 kHz mono and uploaded so the dashboard's audio player can presign and stream them.

Runnable demo agents

Two examples ship with the SDK so you can see voice observability flow end-to-end without writing any agent code:

# Minimal — connects to LiveKit, attaches observability, greets the participant
python -m ashr_labs.voice_obs.examples.livekit_worker dev

# Full — a more "real-feeling" support agent built on the same primitives
python -m ashr_labs.voice_obs.examples.ashr_support_agent dev

Required env vars:

• LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET
• ASHR_LABS_API_KEY (or ASHR_VOICE_OBS_API_KEY)
• ASHR_VOICE_OBS_TENANT_ID

Voice sessions land in the same Observability panel as text-trace sessions; the dashboard auto-renders turns, transcripts, per-stage cost and latency, mixed-audio replay, and barge-in metrics.

---

Common Patterns

Tagging by deployment / version

Adds dashboard filters so you can compare versions side-by-side:

await client
  .trace("handle-ticket", { tags: [`version:${APP_VERSION}`, `env:${ENV}`] })
  .wrap(async (t) => {
    // ...
  });

Capturing exceptions explicitly

The default behavior is good enough for most cases, but if you want richer error context attached to a span:

const t = trace.span("tool:risky");
try {
  const result = await risky();
  t.end({ output: result });
} catch (e) {
  t.end({
    level: "ERROR",
    statusMessage: `${(e as Error).name}: ${(e as Error).message}`,
    metadata: { recoverable: e instanceof RetryableError },
  });
  throw e;
}

Token usage for streaming responses

Generation accepts usage after the stream completes. Stream first, end last:

await trace.generation("stream-reply", { model: "claude-sonnet-4-6" }).wrap(async (gen) => {
  const chunks: string[] = [];
  let usage;
  for await (const event of stream) {
    chunks.push(event.delta);
    if (event.type === "message_stop") {
      usage = event.usage;
    }
  }
  gen.end({
    output: { content: chunks.join("") },
    usage: { input_tokens: usage.input_tokens, output_tokens: usage.output_tokens },
  });
});

Async

Nothing special — Trace, Span, and Generation flush on a background promise, so await trace.flush() is the only place you ever wait. Use them inside async functions identically:

async function handle(request) {
  await client.trace("handle-request").wrap(async (trace) => {
    await trace.generation("answer", { model: "claude-sonnet-4-6" }).wrap(async (gen) => {
      const response = await llm.generate(/* ... */);
      gen.end({ output: response, usage: { input_tokens: 0, output_tokens: 0 } });
    });
  });
}

High-cardinality metadata

Anything you put in metadata becomes filterable in the dashboard. Use canonical keys (prompt_version, experiment_arm, model_temperature) so the filter UI groups them sensibly.

---

Where to Find Your Data

• Live timeline: lab.ashr.io → Observability → Traces. Filter by user, session, time range, tag, or error level.
• Per-trace detail: Click any trace to see the nested span tree, prompts, completions, token usage, and latency breakdown.
• Analytics: Observability → Analytics. Latency P50/P95, token rollups, model usage, error rates, tool performance, time-series charts.
• Voice sessions: Observability → Voice. Turn timeline, transcripts, per-stage breakdown, mixed-audio replay, barge-in/interrupt metrics.

---

Safety Properties

• Never raises. Every public method catches and logs its own errors. trace.end() is non-blocking and never raises; if the flush fails, await trace.flush() resolves to an error object. Spans never throw into your code.
• Never blocks the hot path. trace.end() is enqueue-and-return; the HTTP flush happens on a background promise.
• Bounded memory. Each trace holds at most 10k observations. If your agent emits more than that without ending the trace (extremely rare), the oldest are dropped, a one-time warning is logged, and a dropped_observations count is attached to the trace so the drop is visible in the dashboard.
• Monotonic clocks for duration_ms so durations stay accurate across NTP adjustments. Wall clock is only used for the start_time/end_time timestamps sent over the wire.
• 5-second graceful drain at process exit: un-ended traces are flushed and the background queue is drained (best-effort, up to 5 seconds) before the process exits.

---

Next Steps

• Browse the API Reference for the complete method signatures and parameter tables.
• See Examples for full end-to-end patterns including error tracking and analytics dashboards.
• Read Authentication if you need to manage API keys or rotate credentials.

If you hit issues, the SDK never crashes your agent — but it does log warnings to stderr. Tracing warnings are prefixed with [ashr_labs] so you can spot them during development.