Skip to main content

Quick Start Guide

This guide will help you get started with the Ashr Labs SDK in just a few minutes.

For a complete end-to-end walkthrough of testing your agent, see Testing Your Agent.

Step 1: Get Your API Key​

Before using the SDK, you need an API key:

  1. Log in to the Ashr Labs web app
  2. Click API Keys in the sidebar
  3. Click Create New Key
  4. Give it a name and pick an expiration
  5. Copy the key (it starts with tp_) — you won't be able to see it again!

Step 2: Initialize the Client​

import { AshrLabsClient } from "ashr-labs";

// Only need your API key — baseUrl defaults to production
const client = new AshrLabsClient("tp_your_api_key_here");

// Or load from environment variable
// const client = AshrLabsClient.fromEnv(); // reads ASHR_LABS_API_KEY

Step 3: List Datasets​

// tenantId is auto-resolved from your API key
const response = await client.listDatasets();
const datasets = response.datasets as Record<string, unknown>[];

console.log(`Found ${datasets.length} datasets:`);
for (const d of datasets) {
  console.log(`  - ${d.name} (ID: ${d.id})`);
}

Step 4: Get a Dataset​

Retrieve a dataset with signed URLs for downloading media files:

const dataset = await client.getDataset(42, true, 3600);

console.log(`Dataset: ${dataset.name}`);

// Access test scenarios
const source = dataset.dataset_source as Record<string, unknown>;
const runs = (source.runs ?? {}) as Record<string, Record<string, unknown>>;
for (const [runId, scenario] of Object.entries(runs)) {
  const actions = (scenario.actions ?? []) as unknown[];
  console.log(`  Scenario: ${scenario.title} (${actions.length} actions)`);
}

Step 5: Run an Eval with EvalRunner​

The easiest way to test your agent. Any object with respond() and reset() methods works:

import { EvalRunner } from "ashr-labs";

// Fetch a dataset and run your agent against it
const runner = await EvalRunner.fromDataset(client, 42);
const run = await runner.run(myAgent);

// Or run scenarios in parallel for faster evals
const runParallel = await runner.run(myAgent, { maxWorkers: 4 });

// Inspect metrics
const metrics = run.build().aggregate_metrics as Record<string, unknown>;
console.log(`Passed: ${metrics.tests_passed}/${metrics.total_tests}`);
console.log(`Avg similarity: ${metrics.average_similarity_score}`);

// Submit
await run.deploy(client, 42);

Your agent just needs these two methods:

import type { Agent } from "ashr-labs";

const myAgent: Agent = {
  async respond(message: string) {
    // Call your LLM, return { text: "...", tool_calls: [...] }
    return { text: "response", tool_calls: [] };
  },

  async reset() {
    // Clear conversation history
  },
};

See Testing Your Agent for a full walkthrough with a complete agent example.

Step 6: Build and Deploy Manually (Advanced)​

For custom eval loops, use RunBuilder directly:

import { RunBuilder } from "ashr-labs";

const run = new RunBuilder();
run.start();

const test = run.addTest("bank_analysis");
test.start();

test.addUserText(
  "Please analyze this bank statement.",
  "User asks for analysis",
);
test.addToolCall(
  { name: "extract_pdf_content", arguments_json: '{"file": "statement.pdf"}' },
  { name: "extract_pdf_content", arguments_json: '{"file": "statement.pdf", "pages": "all"}' },
  "partial",
  "Extra 'pages' argument in actual call",
);
test.addAgentResponse(
  { text: "Based on the bank statement analysis..." },
  { text: "After analyzing the bank statement..." },
  "similar",
  0.89,
);

test.complete();
run.complete();
await run.deploy(client, 42);

Step 7: Submit a Request​

Create a generation request:

const request = await client.createRequest(
  "Audio Generation Request",
  {
    agent: { name: "My Agent", description: "Generates audio" },
    context: { domain: "audio", use_case: "Text to speech" },
  },
);

console.log(`Request #${request.id} created: ${request.request_status}`);

Complete Example​

import { AshrLabsClient, EvalRunner, NotFoundError } from "ashr-labs";

const client = new AshrLabsClient("tp_your_api_key_here");

async function main() {
  // 1. Fetch the dataset and run the eval
  let runner: EvalRunner;
  try {
    runner = await EvalRunner.fromDataset(client, 42);
  } catch (e) {
    if (e instanceof NotFoundError) {
      console.log("Dataset not found!");
      return;
    }
    throw e;
  }

  // 2. Run agent against all scenarios
  const run = await runner.run(myAgent, {
    onScenario: (sid, s) => console.log(`Running: ${s.title ?? sid}`),
  });

  // 3. Inspect results
  const result = run.build();
  const metrics = result.aggregate_metrics as Record<string, unknown>;
  console.log(`Passed: ${metrics.tests_passed}/${metrics.total_tests}`);
  console.log(`Avg similarity: ${metrics.average_similarity_score}`);
  console.log(`Tool divergences: ${metrics.total_tool_call_divergence}`);

  // 4. Submit
  const created = await run.deploy(client, 42);
  console.log(`Run #${created.id} submitted!`);
}

main();

Next Steps​