Skip to main content

Quick Start

From zero to a graded eval run in a few minutes. For the full walkthrough with a real agent, see Testing Your Agent.

1. Get an API key​

  1. Log in at lab.ashr.io
  2. Click API Keys in the sidebar
  3. Click Create New Key, name it, pick an expiration
  4. Copy the key (it starts with tp_) — it's shown only once

2. Initialize the client​

import { AshrLabsClient } from "ashr-labs";

const client = new AshrLabsClient("tp_your_api_key_here");

// Or read ASHR_LABS_API_KEY (and optional ASHR_LABS_BASE_URL) from the environment:
// const client = AshrLabsClient.fromEnv();

baseUrl defaults to production and your tenantId is resolved from the key on first use, so there's nothing else to configure.

3. Bring an agent​

An agent is any object with two methods: respond(message) returning { text, tool_calls }, and reset() to clear state between scenarios.

import type { Agent } from "ashr-labs";

const myAgent: Agent = {
  async respond(message: string) {
    // call your LLM, collect any tool calls it made
    return { text: "...", tool_calls: [] };
  },
  async reset() {
    // clear conversation history
  },
};

If you're wrapping an Anthropic or OpenAI tool-calling loop, skip the object. Define tools with tool() and use an adapter — it runs the tool loop and accumulates tool_calls for you:

import { AnthropicAgent, tool } from "ashr-labs";

const lookupOrder = tool({
  name: "lookup_order",
  description: "Look up the status of a customer order.",
  parameters: {
    type: "object",
    properties: { order_id: { type: "string" } },
    required: ["order_id"],
  },
  fn: ({ order_id }) => shop.getOrder(order_id as string),
});

const agent = new AnthropicAgent({
  model: "claude-sonnet-4-6",
  system: "You are a support agent.",
  tools: [lookupOrder],
});

OpenAIAgent is the same for OpenAI, and FunctionAgent wraps any other framework — see API Reference → Agent adapters. (AnthropicAgent needs npm install @anthropic-ai/sdk; OpenAIAgent needs npm install openai.)

4. Find a dataset​

const response = await client.listDatasets();
for (const d of response.datasets as Record<string, unknown>[]) {
  console.log(`${d.id}: ${d.name}`);
}

A note on dataset IDs​

The dashboard displays dataset IDs in hexadecimal — a dataset shown as #00000332 is the number 818 in the API (0x332 === 818). The SDK always takes the plain number:

const datasetId = 0x332; // === 818, the dataset shown as #00000332
// const datasetId = parseInt("00000332", 16); // same thing, if copying the string

Passing the decimal value 332 will not error — it just targets a different dataset (whatever row 332 happens to be). When in doubt, confirm the name:

console.log((await client.getDataset(818)).name);

5. Run the eval​

import { EvalRunner } from "ashr-labs";

const runner = await EvalRunner.fromDataset(client, 818);
const run = await runner.run(agent); // runs your agent against every scenario

run() returns a RunBuilder with every scenario recorded. Inspect it locally before sending anything:

const m = run.build().aggregate_metrics as Record<string, number>;
console.log(`${m.total_tests} ${m.tests_passed} ${m.tests_failed}`);

Local metrics reflect what the builder recorded. The authoritative pass/fail verdicts come from server-side grading after you deploy (next step).

6. Deploy and wait for grading​

const created = await run.deploy(client, 818); // grading runs async, server-side
console.log(`Run #${created.id} submitted`);

// Grading is asynchronous (1-3 min). pollRun blocks until it finishes.
// Pass agentId (see step 7) so the attached deeplink opens the run drawer.
const graded = await client.pollRun(created.id as number, { agentId: myAgentId });
const gm = (graded.result as Record<string, unknown>)
  .aggregate_metrics as Record<string, number>;
console.log(`Passed: ${gm.tests_passed}/${gm.total_tests}`);
console.log(`View: ${graded.deeplink}`);

Steps 5 and 6 collapse into one call with runAndDeploy:

const created = await runner.runAndDeploy(agent, client, 818, { agentId: myAgentId });
const graded = await client.pollRun(created.id as number, { agentId: myAgentId });

If you don't want to block, skip pollRun and re-fetch later with client.getRun(runId). Either way, the graded.deeplink field is built with agentId so it opens the run directly. To build a link yourself: client.deeplink(818, { runId: created.id as number, agentId: myAgentId }). See API Reference → deeplink.

7. Group runs under an agent (optional)​

If you eval the same agent across many datasets, create an agent to keep your datasets and runs organized under one record. The agent's configuration lives in config.form_data — a saved dataset-generation preset (the agent block, tools, and context) the dashboard uses to pre-fill new requests. Resolve a stable agent ID once, then key everything off it:

// getOrCreateAgent is idempotent — safe to call on every run.
const agentRecord = await client.getOrCreateAgent(
  "ShopWave Support Agent",
  "Handles orders, refunds, and stock questions",
  {
    form_data: {
      agent: {
        system_prompt: "You are a support agent.",
        tools: [lookupOrder.toAshr()],
      },
      context: { domain: "ecommerce", use_case: "Order and refund support" },
    },
  },
);
const myAgentId = agentRecord.id as number;

await runner.runAndDeploy(agent, client, 818, { agentId: myAgentId });

config is an AgentConfig; form_data is the saved preset that pre-fills new requests (see API Reference → createAgent). tool_definitions/behavior_rules may also be stored as optional descriptive metadata. Datasets and runs linked via agentId are grouped under the agent in the dashboard.

Complete example​

import { AshrLabsClient, AnthropicAgent, EvalRunner, tool, NotFoundError } from "ashr-labs";

const client = AshrLabsClient.fromEnv();

const lookupOrder = tool({
  name: "lookup_order",
  description: "Look up the status of a customer order.",
  parameters: {
    type: "object",
    properties: { order_id: { type: "string" } },
    required: ["order_id"],
  },
  fn: ({ order_id }) => shop.getOrder(order_id as string),
});

const agent = new AnthropicAgent({
  model: "claude-sonnet-4-6",
  system: "You are a support agent.",
  tools: [lookupOrder],
});

async function main() {
  let runner: EvalRunner;
  try {
    runner = await EvalRunner.fromDataset(client, 818);
  } catch (e) {
    if (e instanceof NotFoundError) {
      console.log("Dataset not found");
      return;
    }
    throw e;
  }

  const created = await runner.runAndDeploy(agent, client, 818, {
    onScenario: (sid, s) => console.log(`Running: ${s.title ?? sid}`),
  });
  console.log(`Run #${created.id} submitted`);
}

main();

Next steps​