Skip to main content

Quick Start

From zero to a graded eval run in a few minutes. For the full walkthrough with a real agent, see Testing Your Agent.

Fastest path: let your coding agent set it up​

If you use Claude Code, Cursor, or Codex, you don't have to wire any of this up by hand. Run the bootstrapper once:

npx ashr-labs

That drops a setup-ashr skill into your agent's skill directories (.claude/skills/, .cursor/skills/, .agents/skills/). Open your coding agent and run the setup-ashr skill. It walks through onboarding for you:

  1. Detects your project — language, agent entrypoint, and agent class (by reading your code, not guessing).
  2. Validates your API key against the live API and saves it to .env.
  3. Scaffolds everything — .ashr.json, a CLAUDE.md section, and test-agent + improve-agent skills for whichever agents you use.
  4. Runs one real eval so you see a graded result and a dashboard link before it's done.

From then on, run the test-agent skill (or /test-agent in Claude Code) after any change to your agent. Already have a key handy? npx ashr-labs tp_your_key stashes it in .env first.

Prefer to set it up yourself? The manual steps below do the same thing.

1. Get an API key​

  1. Log in at lab.ashr.io
  2. Click API Keys in the sidebar
  3. Click Create New Key, name it, pick an expiration
  4. Copy the key (it starts with tp_) — it's shown only once

2. Initialize the client​

from ashr_labs import AshrLabsClient

client = AshrLabsClient(api_key="tp_your_api_key_here")

# Or read ASHR_LABS_API_KEY (and optional ASHR_LABS_BASE_URL) from the environment:
# client = AshrLabsClient.from_env()

base_url defaults to production and your tenant_id is resolved from the key on first use, so there's nothing else to configure.

3. Bring an agent​

An agent is any object with two methods: respond(message) returning {"text": ..., "tool_calls": [...]}, and reset() to clear state between scenarios.

class MyAgent:
    def respond(self, message: str) -> dict:
        # call your LLM, collect any tool calls it made
        return {"text": "...", "tool_calls": []}

    def reset(self) -> None:
        self.history = []

If you're wrapping an Anthropic or OpenAI tool-calling loop, skip the class entirely. Define tools with @tool and use an adapter — it runs the tool loop and accumulates tool_calls for you:

from ashr_labs import AnthropicAgent, tool

@tool
def lookup_order(order_id: str) -> dict:
    """Look up the status of a customer order.

    Args:
        order_id: The order ID (e.g. ORD-12345).
    """
    return shop.get_order(order_id)

agent = AnthropicAgent(model="claude-sonnet-4-6", system="You are a support agent.", tools=[lookup_order])

OpenAIAgent is the same for OpenAI, and FunctionAgent wraps any other framework — see API Reference → Agent adapters.

4. Find a dataset​

datasets = client.list_datasets()
for d in datasets["datasets"]:
    print(f"{d['id']}: {d['name']}")

A note on dataset IDs​

The dashboard displays dataset IDs in hexadecimal — a dataset shown as #00000332 is integer 818 in the API (0x332 == 818). The SDK always takes the plain integer:

dataset_id = 0x332          # == 818, the dataset shown as #00000332
# dataset_id = int("00000332", 16)   # same thing, if you're copying the string

Passing the decimal value 332 will not error — it just targets a different dataset (whatever row 332 happens to be). When in doubt, confirm the name:

print(client.get_dataset(818)["name"])

5. Run the eval​

from ashr_labs import EvalRunner

runner = EvalRunner.from_dataset(client, dataset_id=818)
run = runner.run(agent)                         # runs your agent against every scenario

run() returns a RunBuilder with every scenario recorded. Inspect it locally before sending anything:

m = run.build()["aggregate_metrics"]
print(m["total_tests"], m["tests_passed"], m["tests_failed"])

Local metrics reflect what the builder recorded. The authoritative pass/fail verdicts come from server-side grading after you deploy (next step).

6. Deploy and wait for grading​

created = run.deploy(client, dataset_id=818)    # grading runs async, server-side
print(f"Run #{created['id']} submitted")

graded = client.poll_run(created["id"])         # blocks until grading finishes (1–3 min)
m = graded["result"]["aggregate_metrics"]
print(f"Passed: {m['tests_passed']}/{m['total_tests']}")

To get a clickable link into the dashboard, build one with deeplink. Pass agent_id (see step 7) — the analysis view only opens the run drawer once an agent is selected:

print(client.deeplink(818, run_id=created["id"], agent_id=my_agent_id))

The graded["deeplink"] field is also populated, but it omits agent_id, so it lands on the agents list rather than the run. See API Reference → deeplink.

Steps 5 and 6 collapse into one call with run_and_deploy:

created = runner.run_and_deploy(agent, client, dataset_id=818)

7. Group runs under an agent (optional)​

If you eval the same agent across many datasets, create an agent to keep your datasets and runs organized under one record. The agent's configuration lives in config.form_data — a saved dataset-generation preset (the agent block, tools, and context) the dashboard uses to pre-fill new requests. Resolve a stable agent ID once, then key everything off it:

# get_or_create_agent is idempotent — safe to call on every run.
agent_id = client.get_or_create_agent(
    "ShopWave Support Agent",
    description="Handles orders, refunds, and stock questions",
    config={
        "form_data": {
            "agent": {
                "system_prompt": "You are a support agent.",
                "tools": [lookup_order.to_ashr()],
            },
            "context": {"domain": "ecommerce", "use_case": "Order and refund support"},
        },
    },
)["id"]

runner.run_and_deploy(agent, client, dataset_id=818, agent_id=agent_id)

config is an AgentConfig; form_data is the saved preset that pre-fills new requests (see API Reference → create_agent). tool_definitions/behavior_rules may also be stored as optional descriptive metadata. Datasets and runs linked via agent_id are grouped under the agent in the dashboard.

Complete example​

from ashr_labs import AshrLabsClient, AnthropicAgent, EvalRunner, tool, NotFoundError

client = AshrLabsClient.from_env()

@tool
def lookup_order(order_id: str) -> dict:
    """Look up the status of a customer order."""
    return shop.get_order(order_id)

agent = AnthropicAgent(model="claude-sonnet-4-6", system="You are a support agent.", tools=[lookup_order])

def main() -> None:
    try:
        runner = EvalRunner.from_dataset(client, dataset_id=818)
    except NotFoundError:
        print("Dataset not found")
        return

    created = runner.run_and_deploy(
        agent, client, dataset_id=818,
        on_scenario=lambda sid, s: print(f"Running: {s.get('title', sid)}"),
    )
    graded = client.poll_run(created["id"])
    m = graded["result"]["aggregate_metrics"]
    print(f"Passed: {m['tests_passed']}/{m['total_tests']}")

if __name__ == "__main__":
    main()

Next steps​