Ashr Labs Python SDK

Test your AI agent against real scenarios, then watch it in production. One SDK, one API key, two products:

• Testing Platform — generate datasets, run your agent against every scenario offline, compare expected vs. actual behavior, and submit results for server-side grading. This is the EvalRunner / @tool / agent-adapter surface, and it's where most people start.
• Observability — trace your agent's behavior in production (LLM calls, tool invocations, latency, errors). This is the client.trace() surface. Traces render in the Observability panel of the dashboard. Requires the observability feature flag on your tenant.

The two are independent. You can run evals without ever touching tracing, and vice versa.

Install

pip install ashr-labs

Python 3.10+. See Installation for dependency details.

60-second example

Define your agent:

from ashr_labs import AnthropicAgent, tool

@tool
def lookup_order(order_id: str) -> dict:
    """Look up the status of a customer order.

    Args:
        order_id: The order ID (e.g. ORD-12345).
    """
    return shop.get_order(order_id)

agent = AnthropicAgent(model="claude-fable-5", system=SYSTEM_PROMPT, tools=[lookup_order])

Run it against a dataset:

from ashr_labs import AshrLabsClient, EvalRunner

client = AshrLabsClient(api_key="tp_your_api_key_here")

runner = EvalRunner.from_dataset(client, dataset_id=818)
created = runner.run_and_deploy(agent, client, dataset_id=818)

graded = client.poll_run(created["id"])
m = graded["result"]["aggregate_metrics"]
print(f"Passed: {m['tests_passed']}/{m['total_tests']}")

Any object with respond() and reset() works as an agent — see Testing Your Agent.

Dataset IDs are shown in hex on the dashboard. #00000332 is integer 818 — pass dataset_id=818. See Quick Start.

Where to go next

You want to…	Read
Get an API key and make your first call	Quick Start
Install and verify the SDK	Installation
Run a full eval end to end	Testing Your Agent — start here for evals
Cut your setup code with @tool + adapters	API Reference → Tools / Agent adapters
Trace your agent in production	Observability
Send your own OpenTelemetry traces/logs	OpenTelemetry Integration
Look up an exact method signature	API Reference
Handle errors	Error Handling
Authenticate / manage API keys	Authentication
Copy-paste recipes (CI, batch, reporting)	Examples

Advanced surfaces (clearly separate from the core path): VM Integration for browser/desktop agents, Voice Observability for realtime LiveKit voice agents.

Support

Questions or feature requests: support@ashr.io or book a call.