Skip to main content

API Reference

Complete reference for all classes and methods in the Ashr Labs SDK.

The SDK serves two products:

  • Testing Platform — generate eval datasets, run your agent against them, submit graded results. Core methods: createRequest, createRun, EvalRunner, RunBuilder.
  • Observability (separate product) — trace your agent's production behavior (LLM calls, tool invocations, latency, errors). Core methods: trace(), Span, Generation, listObservabilityTraces. Requires the observability feature flag.

These are independent products that share the same SDK and API key.

AshrLabsClient​

The main client class for interacting with the Ashr Labs API.

Constructor​

new AshrLabsClient(
  apiKey: string,
  baseUrl?: string,
  timeout?: number,
)

Parameters:

ParameterTypeRequiredDefaultDescription
apiKeystringYes-Your API key (must start with tp_)
baseUrlstringNoProduction URLBase URL of the API
timeoutnumberNo30Request timeout in seconds

Throws:

  • Error: If the API key format is invalid

Example:

// Minimal — just pass your API key
const client = new AshrLabsClient("tp_your_key_here");

// Custom timeout
const client = new AshrLabsClient("tp_your_key_here", undefined, 60);

fromEnv (static method)​

Create a client from environment variables.

AshrLabsClient.fromEnv(timeout?: number): AshrLabsClient

Reads ASHR_LABS_API_KEY (required) and ASHR_LABS_BASE_URL (optional) from the environment.

Throws:

  • Error: If ASHR_LABS_API_KEY is not set

Example:

// export ASHR_LABS_API_KEY="tp_your_key_here"
const client = AshrLabsClient.fromEnv();

Build a clickable dashboard URL for a dataset, run, or individual scenario. Available both as client.deeplink(...) and as the standalone exported deeplink(...) (identical signature).

deeplink(datasetId: number, opts?: DeeplinkOptions): string

interface DeeplinkOptions {
  runId?: number | null;       // the eval-run row (a specific execution)
  scenarioId?: string | null;  // a per-test test_id, to deep-link one scenario
  agentId?: number | null;     // the agent whose dashboard hosts the dataset drawer
  tab?: string;                // default "analysis"
  base?: string;               // dashboard base; falls back to ASHR_DASHBOARD_URL, then https://lab.ashr.io
}

Pass agentId for any link you expect to click. The analysis tab only restores a dataset / execution deep-link once an agent is selected. Without agentId, the URL lands on the agents list and the run drawer never opens.

This is also why the auto-populated graded["deeplink"] field (added by pollRun / getRun) lands on the agents list: it's built without agentId. For a guaranteed-clickable link, build it yourself with agentId.

Targeting a non-production dashboard: set ASHR_DASHBOARD_URL (e.g. a local dev server), or pass base.

// Open the run drawer for run 11873 on dataset 818, under agent 11:
client.deeplink(818, { runId: 11873, agentId: 11 });
// → https://lab.ashr.io/?tab=analysis&agent=11&dataset=818&execution=11873

// Deep-link a single failed scenario:
client.deeplink(818, { runId: 11873, scenarioId: "abc-123", agentId: 11 });

Session Methods​

init​

Initialize a session and validate authentication.

async init(): Promise<Record<string, unknown>>

Returns: Session information containing user and tenant data

Throws:

  • AuthenticationError: If the API key is invalid or expired

Example:

// Validate credentials and get user/tenant info
const session = await client.init();

const user = session.user as Record<string, unknown>;
const tenant = session.tenant as Record<string, unknown>;
console.log(`User ID: ${user.id}`);
console.log(`Email: ${user.email}`);
console.log(`Tenant ID: ${tenant.id}`);
console.log(`Tenant Name: ${tenant.tenant_name}`);

Dataset Methods​

getDataset​

Retrieve a dataset by ID.

async getDataset(
  datasetId: number,
  includeSignedUrls?: boolean,
  urlExpiresSeconds?: number,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
datasetIdnumberYes-The ID of the dataset (plain number)
includeSignedUrlsbooleanNofalseInclude signed S3 URLs for media
urlExpiresSecondsnumberNo3600URL expiration time in seconds

Dataset IDs are numbers here, hex on the dashboard. A dataset shown as #00000332 in the UI is the number 818 (0x332). Use 0x332 or parseInt("00000332", 16). The decimal value 332 is a different dataset and will not error — confirm with (await client.getDataset(818)).name if unsure.

Returns: The dataset object

Throws:

  • NotFoundError: Dataset not found
  • AuthorizationError: No access to this dataset

Example:

const dataset = await client.getDataset(42, true, 7200);
console.log(dataset.name);

listDatasets​

List datasets for a tenant.

async listDatasets(
  tenantId?: number | null,
  limit?: number,
  cursor?: number | null,
  includeSignedUrls?: boolean,
  urlExpiresSeconds?: number,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
tenantIdnumberNoautoThe tenant ID (auto-resolved if omitted)
limitnumberNo50Maximum results to return
cursornumberNonullPagination cursor — pass next_cursor from the previous response
includeSignedUrlsbooleanNofalseInclude signed S3 URLs
urlExpiresSecondsnumberNo3600URL expiration time

Returns: Object with keys:

  • status: "ok"
  • datasets: Array of dataset objects
  • next_cursor: ID for the next page, or null if no more results

Example:

// tenantId auto-resolved from API key
const response = await client.listDatasets(undefined, 10);
const datasets = response.datasets as Record<string, unknown>[];
for (const dataset of datasets) {
  console.log(`${dataset.id}: ${dataset.name}`);
}

// Pagination
if (response.next_cursor) {
  const nextPage = await client.listDatasets(undefined, 10, response.next_cursor as number);
}

Run Methods​

createRun​

Create a new test run.

async createRun(
  datasetId: number,
  result: Record<string, unknown>,
  tenantId?: number | null,
  runnerId?: number | null,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
datasetIdnumberYes-The dataset ID
resultRecord<string, unknown>Yes-Run results (metrics, status, etc.)
tenantIdnumberNoautoThe tenant ID (auto-resolved if omitted)
runnerIdnumberNonullID of user who ran the test

Returns: The created run object

Example:

const run = await client.createRun(42, {
  status: "passed",
  score: 0.95,
  metrics: {
    accuracy: 0.98,
    latency_ms: 150,
  },
});

getRun​

Retrieve a run by ID.

async getRun(runId: number): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDescription
runIdnumberYesThe run ID

Returns: The run object

Throws:

  • NotFoundError: Run not found

Example:

const run = await client.getRun(99);
const result = run.result as Record<string, unknown>;
console.log(`Score: ${result.score}`);

listRuns​

List runs for a tenant or dataset.

async listRuns(
  datasetId?: number | null,
  tenantId?: number | null,
  limit?: number,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
datasetIdnumberNonullFilter by dataset
tenantIdnumberNoautoFilter by tenant (auto-resolved if omitted)
limitnumberNo50Maximum results

Returns: Object with keys:

  • status: "ok"
  • runs: Array of run objects

Example:

// Get runs for a specific dataset
const response = await client.listRuns(42);
const runs = response.runs as Record<string, unknown>[];
for (const run of runs) {
  const result = run.result as Record<string, unknown>;
  console.log(`Run #${run.id}: ${result.status}`);
}

deleteRun​

Delete a test run.

async deleteRun(runId: number): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDescription
runIdnumberYesThe run ID to delete

Returns: Confirmation of deletion

Throws:

  • NotFoundError: Run not found

Example:

await client.deleteRun(99);
console.log("Run deleted");

Observability — Production Agent Tracing​

This is a separate product from the Testing Platform. The testing platform (datasets, eval runs, RunBuilder, EvalRunner) is for offline evaluation. Observability is for tracing your agent in production. They share the same SDK and API key but are independent features.

Trace your agent's production behavior — LLM calls, tool invocations, retrieval steps, guardrail checks, and more. Requires the observability feature flag to be enabled for your tenant.

Production-safe: tracing never throws or interferes with your agent, and never blocks your hot path. trace.end() enqueues the trace on a background thread and returns immediately; the HTTP flush runs in the background. If the backend is unreachable the failure is logged and surfaced via await trace.flush() (which resolves with an error object), never thrown. Un-ended traces and dangling spans are drained on process exit; per-trace buffers are bounded at 10k observations; duration_ms uses a monotonic clock.

client.trace​

Start a new trace for a production agent interaction.

const trace = client.trace(
  name: string,
  opts?: {
    userId?: string;
    sessionId?: string;
    metadata?: Record<string, unknown>;
    tags?: string[];
  },
): Trace

Parameters:

ParameterTypeRequiredDescription
namestringYesName for this trace (e.g. "handle-ticket")
opts.userIdstringNoEnd-user ID for grouping
opts.sessionIdstringNoConversation/session ID
opts.metadataRecordNoArbitrary metadata
opts.tagsstring[]NoTags for filtering

Returns: A Trace instance. Supports the wrap() pattern.

Trace methods​

MethodDescription
trace.span(name, opts?)Create a top-level span
trace.generation(name, opts?)Create a top-level generation (LLM call)
trace.event(name, opts?)Record a point-in-time event
trace.wrap(fn)Run callback, enqueue flush on completion
trace.end(opts?)Enqueue the trace for a background flush. Non-blocking, never throws. Returns { status, trace_id }.
await trace.flush()Block until the background flush resolves; returns the backend response (or error object), or null if end() was never called.
trace.traceIdServer-assigned trace ID (available after await trace.flush())

Span methods​

MethodDescription
span.span(name, opts?)Create a child span
span.generation(name, opts?)Create a child generation
span.event(name, opts?)Record an event under this span
span.wrap(fn)Run callback, auto-end on completion
span.end(opts?)Mark the span as complete

If wrap() callback throws, the span auto-ends with level: "ERROR" and the exception message captured in statusMessage.

Generation methods​

Inherits all Span methods, plus:

MethodDescription
gen.end(opts?)Mark complete. Accepts output, usage: { input_tokens, output_tokens }, statusMessage, level.

wrap() ensures spans/traces are always ended, even if your code throws:

await trace.wrap(async (t) => {
  await t.span("tool:search", { input: { q: "..." } }).wrap(async (s) => {
    const data = await search(...);
    s.end({ output: data });
    return data;
  });

  const gen = t.generation("respond", { model: "claude-sonnet-4-6" });
  const response = await callLlm(...);
  gen.end({ output: response, usage: { input_tokens: 100, output_tokens: 50 } });
});
// trace.end() called automatically

Manual instrumentation​

const trace = client.trace("support-chat", { userId: "user_42", sessionId: "conv_abc" });

const gen = trace.generation("classify", { model: "claude-sonnet-4-6",
  input: [{ role: "user", content: "Reset my password" }] });
gen.end({ output: { intent: "password_reset" },
  usage: { input_tokens: 50, output_tokens: 12 } });

const tool = trace.span("tool:reset_password", { input: { user_id: "42" } });
tool.end({ output: { success: true } });

trace.event("guardrail-check", { input: { passed: true } });

trace.end({ output: { resolution: "password_reset_complete" } }); // enqueues, returns immediately
await trace.flush();                                              // optional: wait for the backend
console.log(trace.traceId);

listObservabilityTraces​

List traces for the current tenant.

await client.listObservabilityTraces(opts?: {
  userId?: string;
  sessionId?: string;
  limit?: number;
  page?: number;
}): Promise<{ status: string; traces: object[]; total: number }>
ParameterTypeRequiredDefaultDescription
opts.userIdstringNoundefinedFilter by end-user
opts.sessionIdstringNoundefinedFilter by session
opts.limitnumberNo50Max results per page (max 100)
opts.pagenumberNo1Page number

Returns: { status: "ok", traces: [...], total: number }

getObservabilityTrace​

Get a single trace with its full observation tree.

await client.getObservabilityTrace(traceId: string): Promise<{ status: string; trace: object }>

Returns: { status: "ok", trace: {...} } — the trace includes an observations list with id, name, type, parent_observation_id, input, output, metadata, model, usage, level, start_time, end_time.

getObservabilityAnalytics​

Get analytics overview for the current tenant.

await client.getObservabilityAnalytics(days?: number): Promise<{
  status: string;
  overview: { total_traces, avg_latency_ms, total_input_tokens, total_output_tokens, error_rate, total_tool_calls, ... };
  tool_performance: { tool_name, total_calls, error_rate, avg_latency_ms }[];
  model_usage: { model, total_calls, total_tokens, avg_latency_ms }[];
}>

Overview includes: total_traces, avg_latency_ms, p95_latency_ms, total_input_tokens, total_output_tokens, error_rate, total_tool_calls, unique_users, unique_sessions.

getObservabilityErrors / getObservabilityToolErrors​

await client.getObservabilityErrors(opts?: { days?: number; limit?: number; page?: number })
await client.getObservabilityToolErrors(opts?: { days?: number; limit?: number; page?: number })

Returns: { status: "ok", traces: [...], total: number } — traces with errors or tool failures, most recent first.

SDK Notes — Platform Advisories​

SDK Notes are platform advisories delivered to your SDK from Ashr Labs. They communicate context changes, best practices, deprecations, or breaking changes that may affect how you configure or run your agent.

Notes are automatically fetched when the client initializes (via init()). You can also refresh them on demand.

client.notes (getter)​

Get cached SDK notes from the last init() or getNotes() call. No network request is made.

get notes(): Record<string, unknown>[]

Returns: List of active notes for your tenant.

Example:

const client = new AshrLabsClient("tp_...");
await client.init();

// Notes are auto-fetched on init
for (const note of client.notes) {
  console.log(`[${note.severity}] ${note.title}: ${note.content}`);
}

getNotes​

Fetch fresh SDK notes from the platform. Updates the cached client.notes.

async getNotes(agentId?: number | null): Promise<Record<string, unknown>[]>

Parameters:

ParameterTypeRequiredDefaultDescription
agentIdnumber | nullNoundefinedInclude notes targeted at this specific agent

Returns: List of active notes (global + tenant-specific, plus agent-specific if agentId is provided).

Example:

// Refresh notes
const notes = await client.getNotes();

// Filter by agent
const notes = await client.getNotes(42);

// Check for breaking changes
const breaking = notes.filter(n => n.category === "breaking_change");
if (breaking.length) {
  console.log("Warning: Breaking changes detected:");
  for (const n of breaking) {
    console.log(`  ${n.title}: ${n.content}`);
  }
}

Note categories: info, warning, breaking_change, best_practice, deprecation

Severity levels: info, warning, critical

Request Methods​

createRequest​

Create a dataset generation request.

async createRequest(
  requestName: string,
  request: Record<string, unknown>,
  requestInputSchema?: Record<string, unknown> | null,
  tenantId?: number | null,
  requestorId?: number | null,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
requestNamestringYes-Name/title for the request
requestRecord<string, unknown>Yes-The generation config (see below)
requestInputSchemaRecord<string, unknown>NoautoJSON Schema for validating the request. A permissive default is sent if omitted. If your agent has tools, include them here under the "tools" key so they're auto-saved as skill templates.
tenantIdnumberNoautoThe tenant ID (auto-resolved if omitted)
requestorIdnumberNoautoID of requesting user (auto-resolved if omitted)

Returns: The created request object

Generation config structure (the request object):

The config has two required sections (agent and context) and several optional sections.

metadata (optional)​

FieldTypeDescription
dataset_namestringName for the generated dataset
descriptionstringDescription of what this dataset tests

agent (required)​

At least one of name, description, or system_prompt is required.

FieldTypeDefaultDescription
namestring-Agent name
descriptionstring-What the agent does
system_promptstring-System prompt given to the agent
toolsobject[][]Tools the agent can call (see below)
accepted_inputsobjecttext onlyInput modalities (see below)
output_formatobject{ type: "text" }"text" or "structured" with optional schema
input_schemaobject-Custom structured input schema (see below)

Tool definition:

{
  name: "tool_name",              // snake_case tool name
  description: "What it does",     // Used by test generator for realistic scenarios
  parameters: {                    // JSON Schema for tool parameters
    type: "object",
    properties: {
      arg_name: { type: "string", description: "What this arg is" },
    },
    required: ["arg_name"],
  },
  returns: {                       // (optional) Return value schema
    type: "object",
    description: "What the tool returns",
  },
}

Accepted inputs — values can be boolean or { enabled: boolean }:

accepted_inputs: {
  text: true,            // (default true) Text input
  audio: false,          // Audio: mp3, wav, m4a, ogg, webm
  file: false,           // Files: pdf, txt, csv, json, xml, html, md, docx, xlsx
  image: false,          // Images: jpg, png, gif, webp
  video: false,          // Video: mp4, webm, mov, avi
  conversation: false,   // Multi-participant conversations with inferred roles
}

Input schema — define structured data users provide to the agent:

input_schema: {
  name: "OrderInput",
  description: "Data the customer provides",
  fields: [
    { name: "order_id", type: "string", description: "Order ID", required: true },
    { name: "priority", type: "string", description: "Priority level", enum: ["low", "medium", "high"] },
  ],
  example: { order_id: "ORD-123", priority: "high" },
}

context (required)​

At least one of domain, use_case, or scenario_context is required.

FieldTypeDefaultDescription
domainstring-Domain: "banking", "healthcare", "e-commerce", "legal", "education", "customer_service", "technology", "travel", "insurance", "other"
use_casestring-Specific use case description (min 10 chars recommended)
scenario_contextstring-Additional scenario context
user_personaobject-{ type: string, description: string } — who interacts with the agent
sample_dataobject-Free-form mock environment (names, IDs, dates, records) made available to test generation and to the grader. See Sample data: mocking the agent's environment below.
Sample data: mocking the agent's environment​

context.sample_data is the way to give the platform a stand-in for the runtime data the agent would normally read from its environment — customer records, calendar entries, inventory rows, account balances, anything the agent might otherwise have to fetch via a tool. The shape is intentionally free-form (an object / array / nested structure — usually under an examples key, but anything JSON-serializable works) so you can describe the world the way it makes sense for your domain.

It feeds two places:

  1. Test generation. The analysis and generation sub-agents see sample_data and ground the synthesized intents in the entities you provide, so the generated scenarios reference the actual names / IDs / dates in your mock environment instead of inventing fresh ones.

  2. Grading. When the LLM judge evaluates tool-call arguments, NOT_CALLED recoveries, and text responses, it receives sample_data as a "Mock Environment Data" block. If a divergence between the expected and actual tool args is explained by the mock environment (e.g. the agent referenced an entity that exists there, or served a read-only answer from environment knowledge instead of calling a fetch tool), the judge can downgrade severity rather than failing the test outright. Conversely, agents that invent entities contradicting the mock environment are flagged as factually wrong.

This is the right place to use sample_data if your concern is "the tool call would have failed because the agent didn't have access to this data, but the data should clearly have been available."

Example:

context: {
  domain: "healthcare",
  use_case: "Front-desk agent confirming appointments",
  sample_data: {
    examples: [
      { patient: "Jane Doe",   dob: "1984-03-12", appt: "2026-05-20T09:00", provider: "Dr. Patel" },
      { patient: "Marcus Lee", dob: "1971-11-04", appt: "2026-05-20T10:30", provider: "Dr. Patel" },
    ],
    clinic_hours: "Mon–Fri 8am–5pm",
  },
},

Notes:

  • There's no strict schema. The judge LLM reads it as JSON and uses context clues, so put what would actually be visible to the agent.
  • sample_data is persisted on dataset_source.sample_data for the generated dataset, so it's available to every run/regrade pass without needing to be re-supplied.
  • Use this for environment state, not behavioral instructions. Things like "always greet politely" belong in expected_behaviors.

test_config (optional)​

FieldTypeDefaultDescription
num_variationsnumber5Number of test scenarios (1-50)
strategystring"balanced""focused", "diverse", or "balanced"
coverageobjectall true{ happy_path: boolean, edge_cases: boolean, error_handling: boolean, boundary_values: boolean }
complexity_distributionobjectauto{ simple: 0.3, moderate: 0.5, complex: 0.2 } — must sum to ~1.0
focus_areasstring[][]Specific areas to focus testing on
excludestring[][]Scenarios or test types to exclude

generation_options (required)​

FieldTypeDefaultDescription
generate_audiobooleanfalseGenerate audio test inputs
generate_filesbooleanfalseGenerate file test inputs (PDF, CSV, etc.)
generate_imagesbooleanfalseGenerate image test inputs
generate_videosbooleanfalseGenerate video test inputs
generate_simulationsbooleanfalseGenerate website session replay simulation videos

Example:

const req = await client.createRequest(
  "Support Agent Eval",
  {
    metadata: { dataset_name: "Support Eval" },
    agent: {
      name: "Support Bot",
      description: "Answers customer questions",
      system_prompt: "You are a helpful support agent.",
      tools: [{ name: "lookup_order", description: "Look up an order",
                parameters: { type: "object", properties: { order_id: { type: "string" } }, required: ["order_id"] } }],
      accepted_inputs: { text: true, audio: false, file: false, image: false, video: false },
      output_format: { type: "text" },
    },
    context: { domain: "ecommerce", use_case: "Customers asking about orders", scenario_context: "Online store" },
    test_config: { num_variations: 3, coverage: { happy_path: true, edge_cases: true } },
    generation_options: { generate_audio: false, generate_files: false, generate_simulations: false },
  },
);

// Use waitForRequest or generateDataset instead of manual polling
const completed = await client.waitForRequest(req.id as number);
console.log(`Status: ${completed.request_status}`);

getRequest​

Retrieve a request by ID.

async getRequest(requestId: number): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDescription
requestIdnumberYesThe request ID

Returns: The request object

Throws:

  • NotFoundError: Request not found

Example:

const req = await client.getRequest(123);
console.log(`Status: ${req.request_status}`);

listRequests​

List requests for a tenant.

async listRequests(
  tenantId?: number | null,
  status?: string | null,
  limit?: number,
  cursor?: number | null,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
tenantIdnumberNoautoThe tenant ID (auto-resolved if omitted)
statusstringNonullFilter by status
limitnumberNo50Maximum results
cursornumberNonullPagination cursor — pass next_cursor from the previous response

Returns: Object with keys:

  • status: "ok"
  • requests: Array of request objects

Example:

// Get pending requests
const response = await client.listRequests(undefined, "pending");
const requests = response.requests as Record<string, unknown>[];
for (const req of requests) {
  console.log(`Request #${req.id}: ${req.request_name}`);
}

Agent Methods​

Agents group datasets and define grading behavior. Each dataset can belong to one agent.

listAgents​

List all agents for your tenant with dataset counts.

async listAgents(): Promise<Record<string, unknown>[]>

Returns: List of agent objects with id, name, description, config, dataset_count.

Example:

const agents = await client.listAgents();
for (const agent of agents) {
  console.log(`${agent.name}: ${agent.dataset_count} datasets`);
}

createAgent​

Create a new agent.

async createAgent(
  name: string,
  description?: string | null,
  config?: Record<string, unknown> | null,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
namestringYes-Agent name (unique per tenant)
descriptionstringNonullWhat this agent does
configRecord<string, unknown>NonullThe agent's configuration (an AgentConfig). Its form_data is the agent's saved generation preset; tool_definitions / behavior_rules are optional descriptive metadata. See below.

Config structure — an AgentConfig:

KeyTypeMeaning
form_dataobjectThe consumed field. The agent's saved dataset-generation preset — a full generation config (same shape as generateDataset's config: agent / context / test_config / …). The dashboard pre-fills new dataset requests for this agent from it, so the agent's tools live at form_data.agent.tools.
tool_definitionsToolDefinition[]Optional descriptive metadata — each { name, description, required }. Stored on the record; not currently read by the platform.
behavior_rulesBehaviorRule[]Optional descriptive metadata — each { rule, strictness }. Stored on the record; not currently read by the platform.
{
  // The consumed field — a saved generation preset that pre-fills new requests.
  form_data: {
    agent: {
      system_prompt: "You are a helpful support agent for ShopWave.",
      tools: [
        { name: "lookup_order", description: "Look up an order",
          parameters: { type: "object",
                        properties: { order_id: { type: "string" } },
                        required: ["order_id"] } },
      ],
    },
    context: { domain: "ecommerce", use_case: "Order and refund support" },
  },
  // Optional descriptive metadata (stored, not yet consumed):
  tool_definitions: [
    { name: "lookup_order", required: true },
    { name: "end_session", required: false },
  ],
}

Example:

const agent = await client.createAgent("Support Bot", "Healthcare scheduling agent", {
  form_data: {
    agent: {
      system_prompt: "You are a helpful support agent for ShopWave.",
      tools: [lookupOrder.toAshr(), processRefund.toAshr()],
    },
    context: { domain: "ecommerce", use_case: "Order and refund support" },
  },
});
console.log(`Created agent: ${agent.id}`);

getOrCreateAgent​

Return an existing agent by name (case-insensitive match), or create it if none exists. This is the safe way to obtain a stable agentId on every run without hitting duplicate-name errors.

async getOrCreateAgent(
  name: string,
  description?: string | null,
  config?: Record<string, unknown> | null,
): Promise<Record<string, unknown>>

description and config are used only when creating. The config is an AgentConfig — the form_data generation preset, plus optional tool_definitions/behavior_rules; see createAgent. Returns the agent record (an object with id, name, description, config, …).

Example:

const agentId = (await client.getOrCreateAgent("Support Bot", "Scheduling agent")).id as number;
await runner.runAndDeploy(agent, client, 818, { agentId });

getAgent​

Look up an agent by its stable numeric ID. Prefer this over name-based lookup once you have the ID — names are mutable, IDs are not.

async getAgent(agentId: number): Promise<Record<string, unknown>>

Returns the agent record. Throws: NotFoundError if no agent has that ID.

updateAgent​

Update an agent's name, description, or config.

async updateAgent(
  agentId: number,
  opts?: { name?: string; description?: string; config?: Record<string, unknown> },
): Promise<Record<string, unknown>>

Note: config replaces the entire config object — merge locally before updating if you want to preserve existing fields.

deleteAgent​

Soft-delete an agent. Datasets are unlinked but not deleted.

async deleteAgent(agentId: number): Promise<Record<string, unknown>>

getAgentDatasets​

Get all datasets linked to an agent.

async getAgentDatasets(
  agentId: number,
  extraParams?: Record<string, unknown>,
): Promise<Record<string, unknown>>

Returns: Object with agent (the agent object) and datasets (list of dataset objects).

setDatasetAgent​

Assign or unassign an agent to a dataset.

async setDatasetAgent(datasetId: number, agentId: number | null): Promise<Record<string, unknown>>

Pass agentId=null to unlink a dataset from its agent.

API Key Methods​

listApiKeys​

List API keys for your tenant.

async listApiKeys(
  includeInactive?: boolean,
): Promise<Record<string, unknown>[]>

Parameters:

ParameterTypeRequiredDefaultDescription
includeInactivebooleanNofalseInclude revoked keys

Returns: Array of API key objects

Note: For security, only the key prefix is returned, not the full key.

Example:

const keys = await client.listApiKeys();
for (const key of keys) {
  console.log(`${key.key_prefix}... - ${key.name}`);
}

revokeApiKey​

Revoke an API key.

async revokeApiKey(apiKeyId: number): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDescription
apiKeyIdnumberYesThe API key ID to revoke

Returns: Confirmation of revocation

Throws:

  • NotFoundError: API key not found

Example:

await client.revokeApiKey(123);
console.log("API key revoked");

Convenience Methods​

waitForRequest​

Block until a request reaches a terminal state (completed or failed).

async waitForRequest(
  requestId: number,
  timeout?: number,
  pollInterval?: number,
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
requestIdnumberYes-The request ID to poll
timeoutnumberNo600Maximum seconds to wait
pollIntervalnumberNo5Seconds between polls

Returns: The final request object

Throws:

  • Error: If the request doesn't finish within timeout seconds
  • AshrLabsError: If the request fails

Example:

const req = await client.createRequest("My Eval", config);
const completed = await client.waitForRequest(req.id as number, 300);
console.log(`Status: ${completed.request_status}`);

pollRun​

Block until backend grading completes for a run. After deploy(), the backend grades tool arguments and text responses asynchronously (typically 1-3 minutes). This method polls getRun() until aggregate_metrics.tests_passed is populated.

async pollRun(
  runId: number,
  opts?: {
    timeout?: number;        // max seconds to wait (default 300)
    pollInterval?: number;   // seconds between polls (default 20)
    agentId?: number | null; // agent the dataset is grouped under
    onPoll?: (elapsedSeconds: number, run: Record<string, unknown>) => void;
  },
): Promise<Record<string, unknown>>

Pass agentId so the attached deeplink opens the run drawer instead of the agents list (see deeplink).

Returns: The fully graded run object, with deeplink and failed_tests attached.

Throws: Error if grading doesn't finish within timeout seconds.

Example:

const created = await run.deploy(client, 818);
const graded = await client.pollRun(created.id as number, { agentId: 11 });
const m = (graded.result as Record<string, unknown>).aggregate_metrics as Record<string, number>;
console.log(`Passed: ${m.tests_passed}/${m.total_tests} — ${graded.deeplink}`);

generateDataset​

Create a dataset generation request, wait for completion, and fetch the result. Combines createRequest + waitForRequest + getDataset into one call.

Missing context fields (use_case, scenario_context) are auto-filled from the agent's name and description. A default test_config is added if not provided.

There are two ways to call it: pass a full config object, or use the flat form that builds the config for you (and accepts Tool objects directly). Passing both config and any builder field throws; passing neither also throws.

async generateDataset(
  requestName: string,
  options: GenerateDatasetOptions,
): Promise<[number, Record<string, unknown>]>

interface GenerateDatasetOptions {
  // EITHER pass a full config…
  config?: Record<string, unknown>;        // same structure as createRequest's `request`
  // …OR the flat builder fields (never both):
  agent?: string | Record<string, unknown>;
  description?: string;
  systemPrompt?: string;
  tools?: (Tool | Record<string, unknown>)[];  // Tool objects rendered via .toAshr()
  domain?: string;
  useCase?: string;
  scenarioContext?: string;
  numVariations?: number;
  // common options:
  requestInputSchema?: Record<string, unknown> | null;
  timeout?: number;        // default 600
  pollInterval?: number;   // default 5
  agentId?: number | null; // group the resulting dataset under this agent
}

Returns: A tuple [datasetId, datasetSource] where datasetSource is the object containing "runs".

Throws:

  • Error: If generation doesn't finish in time
  • AshrLabsError: If generation fails or no datasets are found

Example — full config form:

const [datasetId] = await client.generateDataset("Support Agent Eval", {
  config: {
    metadata: { dataset_name: "Support Eval" },
    agent: { name: "Support Bot", description: "Handles orders" },
    context: { domain: "ecommerce", use_case: "Order support" },
    test_config: { num_variations: 10, coverage: { happy_path: true } },
    generation_options: { generate_audio: false, generate_files: false },
  },
});

Flat form — the same thing without the nested object. The builder fills in accepted_inputs (text-only) and generation_options (no extra assets) with sensible defaults, back-fills missing context from the agent's name/description, and accepts Tool objects directly (rendered via .toAshr()):

const [datasetId, source] = await client.generateDataset("ShopWave Support Eval", {
  agent: "ShopWave Support Agent",
  systemPrompt: "You are a helpful support agent for ShopWave.",
  tools: [lookupOrder, processRefund],          // Tool objects, or raw dicts
  domain: "ecommerce",
  useCase: "Customers contacting support about orders and refunds",
  numVariations: 25,
});
const runs = (source.runs ?? {}) as Record<string, unknown>;
console.log(`Dataset #${datasetId}: ${Object.keys(runs).length} scenarios`);

Pass either config or these builder fields — passing both throws.

Generating a dataset costs time and money on every run. For day-to-day testing, generate once and re-run with EvalRunner.fromDataset(client, datasetId).

Utility Methods​

healthCheck​

Check if the API is reachable.

async healthCheck(): Promise<Record<string, unknown>>

Returns: Status information

Example:

const status = await client.healthCheck();
console.log(`API Status: ${status.status}`);

RunBuilder​

A builder for incrementally constructing run result objects as an agent executes tests. Once complete, the result can be deployed via the client.

Constructor​

new RunBuilder()

No parameters. Creates a run in "pending" status.

RunBuilder.start​

Mark the run as started. Records the current timestamp.

run.start(): this

Returns: this (for chaining)

RunBuilder.addTest​

Create and register a new test within this run.

run.addTest(testId: string): TestBuilder

Parameters:

ParameterTypeRequiredDescription
testIdstringYesUnique identifier for the test case

Returns: TestBuilder — A builder for the individual test

RunBuilder.complete​

Mark the run as completed. Records the current timestamp.

run.complete(status?: string): this

Parameters:

ParameterTypeRequiredDefaultDescription
statusstringNo"completed"Final status ("completed" or "failed")

Returns: this (for chaining)

RunBuilder.build​

Serialize the full run result to an object.

run.build(): Record<string, unknown>

Returns: An object matching the run result schema, ready to pass to client.createRun(datasetId, result). The aggregate_metrics block is computed automatically and contains:

KeyTypeMeaning
total_testsnumberNumber of tests in the run
tests_passednumberTests with status "completed"
tests_failednumberTests with status "failed"
average_similarity_scorenumber | nullMean semantic similarity across agent responses (null until graded)
total_tool_call_divergencenumberCount of tool calls not matched "exact"
total_response_divergencenumberCount of agent responses not matched "exact"

These are the same keys locally and after server-side grading. Locally they're optimistic (a test counts as passed unless it errored); the grader replaces them with real verdicts after deploy() + pollRun().

RunBuilder.deploy​

Build the result and submit it as a new run via the API.

run.deploy(
  client: AshrLabsClient,
  datasetId: number,
  tenantId?: number,
  runnerId?: number,
  agentId?: number | null,   // auto-links the dataset to this agent before creating the run
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
clientAshrLabsClientYes-An authenticated client instance
datasetIdnumberYes-The dataset this run is for
tenantIdnumberNoautoThe tenant (auto-resolved if omitted)
runnerIdnumberNoundefinedID of the user who ran the test
agentIdnumberNonullAgent to auto-link the dataset to

Returns: The created run object from the API

Example:

import { AshrLabsClient, RunBuilder } from "ashr-labs";

const client = new AshrLabsClient("tp_...");

const run = new RunBuilder();
run.start();

const test = run.addTest("bank_analysis");
test.start();
test.addUserText("Analyze this", "User prompt");
test.addToolCall(
  { name: "analyze", arguments: { data: "input" } },
  { name: "analyze", arguments: { data: "input" } },
  "exact",
);
test.complete();

run.complete();
const createdRun = await run.deploy(client, 42);
console.log(`Run #${createdRun.id} created`);

TestBuilder​

Builds a single test result incrementally. Returned by RunBuilder.addTest().

TestBuilder.start​

Mark the test as started. Records the current timestamp.

test.start(): this

Returns: this (for chaining)

TestBuilder.addUserFile​

Record a user file input action.

test.addUserFile(
  filePath: string,
  description: string,
  actionIndex?: number,
): this

Parameters:

ParameterTypeRequiredDefaultDescription
filePathstringYes-Path to the file in the dataset
descriptionstringYes-Description of the action
actionIndexnumberNoautoExplicit index, or auto-incremented

Returns: this (for chaining)

TestBuilder.addUserText​

Record a user text input action.

test.addUserText(
  text: string,
  description: string,
  actionIndex?: number,
): this

Parameters:

ParameterTypeRequiredDefaultDescription
textstringYes-The user's text input
descriptionstringYes-Description of the action
actionIndexnumberNoautoExplicit index, or auto-incremented

Returns: this (for chaining)

TestBuilder.addToolCall​

Record an agent tool call action with expected vs actual comparison.

test.addToolCall(
  expected: Record<string, unknown>,
  actual: Record<string, unknown>,
  matchStatus: string,
  divergenceNotes?: string | null,
  actionIndex?: number,
  argumentComparison?: Record<string, unknown> | null,
): this

Parameters:

ParameterTypeRequiredDefaultDescription
expectedRecord<string, unknown>Yes-Expected tool call (name, arguments)
actualRecord<string, unknown>Yes-Actual tool call made by the agent
matchStatusstringYes-"exact", "partial", or "mismatch"
argumentComparisonRecord<string, unknown>NonullStructured diff from compareArgsStructural(). Recommended — the backend grader may skip tool calls without it.
divergenceNotesstringNonullNotes explaining the divergence
actionIndexnumberNoautoExplicit index, or auto-incremented

Returns: this (for chaining)

TestBuilder.addAgentResponse​

Record an agent text response with expected vs actual comparison.

test.addAgentResponse(
  expectedResponse: Record<string, unknown>,
  actualResponse: Record<string, unknown>,
  matchStatus: string,
  semanticSimilarity?: number | null,
  divergenceNotes?: string | null,
  actionIndex?: number,
): this

Parameters:

ParameterTypeRequiredDefaultDescription
expectedResponseRecord<string, unknown>Yes-The expected response content
actualResponseRecord<string, unknown>Yes-The actual response from the agent
matchStatusstringYes-"exact", "similar", or "divergent"
semanticSimilaritynumberNonullSimilarity score (0.0 to 1.0)
divergenceNotesstringNonullNotes explaining the divergence
actionIndexnumberNoautoExplicit index, or auto-incremented

Returns: this (for chaining)

TestBuilder.setVmStream​

Attach VM session logs to this test. For agents that operate in a browser or virtual machine.

test.setVmStream(
  provider: string,
  opts?: {
    sessionId?: string;
    durationMs?: number;
    logs?: Record<string, unknown>[];
    metadata?: Record<string, unknown>;
  },
): this

Parameters:

ParameterTypeRequiredDefaultDescription
providerstringYes-VM provider name (e.g. "browserbase", "scrapybara", "steel")
opts.sessionIdstringNo-Provider session ID for linking
opts.durationMsnumberNo-Total session duration in milliseconds
opts.logsRecord[]No-Timestamped log entries (see below)
opts.metadataRecordNo-Additional provider-specific metadata

Log entry format: Each entry should have ts (number, ms offset from start) and type (string):

{ ts: 0, type: "navigation", data: { url: "https://..." } }
{ ts: 1200, type: "action", data: { action: "click", selector: "#btn" } }
{ ts: 3000, type: "error", data: { message: "Element not found" } }

Example:

test.setVmStream("browserbase", {
  sessionId: "sess_abc123",
  durationMs: 12000,
  logs: [
    { ts: 0, type: "navigation", data: { url: "https://app.example.com" } },
    { ts: 2000, type: "action", data: { action: "click", selector: "#submit" } },
    { ts: 5000, type: "network", data: { method: "POST", url: "/api/order", status: 201 } },
  ],
});

Returns: this (for chaining)

TestBuilder.setKernelVm​

Convenience method for attaching a Kernel browser session. Sets provider="kernel" and exposes Kernel-specific metadata fields. Fields map to Kernel's browser API response.

test.setKernelVm(
  sessionId: string,
  opts?: {
    durationMs?: number;
    logs?: Record<string, unknown>[];
    liveViewUrl?: string;
    cdpWsUrl?: string;
    replayId?: string;
    replayViewUrl?: string;
    headless?: boolean;
    stealth?: boolean;
    viewport?: { width: number; height: number };
  },
): this

Parameters:

ParameterTypeRequiredDefaultDescription
sessionIdstringYes-Kernel browser session ID
opts.durationMsnumberNo-Total session duration in milliseconds
opts.logsRecord[]No-Timestamped log entries (same format as setVmStream)
opts.liveViewUrlstringNo-Remote live-view URL (browser_live_view_url)
opts.cdpWsUrlstringNo-Chrome DevTools Protocol WebSocket URL
opts.replayIdstringNo-ID of the session recording
opts.replayViewUrlstringNo-URL to view the session replay
opts.headlessbooleanNo-Whether the session ran in headless mode
opts.stealthbooleanNo-Whether anti-bot stealth mode was enabled
opts.viewportobjectNo-Browser viewport, e.g. { width: 1920, height: 1080 }

Example:

test.setKernelVm("kern_sess_abc123", {
  durationMs: 15000,
  logs: [
    { ts: 0, type: "navigation", data: { url: "https://app.example.com" } },
    { ts: 1200, type: "action", data: { action: "click", selector: "#login" } },
    { ts: 3000, type: "screenshot", data: { s3_key: "vm-streams/.../frame.png" } },
  ],
  replayId: "replay_abc123",
  replayViewUrl: "https://www.kernel.sh/replays/replay_abc123",
  stealth: true,
  viewport: { width: 1920, height: 1080 },
});

Returns: this (for chaining)

TestBuilder.complete​

Mark the test as completed. Records the current timestamp.

test.complete(status?: string): this

Parameters:

ParameterTypeRequiredDefaultDescription
statusstringNo"completed"Final status ("completed" or "failed")

Returns: this (for chaining)

TestBuilder.build​

Serialize this test to an object matching the run result schema.

test.build(): Record<string, unknown>

Returns: An object with test_id, status, action_results, started_at, and completed_at.

EvalRunner​

Runs an agent against every scenario in a dataset and records results. This is the high-level API that encapsulates the full eval loop — iterating scenarios, calling the agent, comparing tool calls and text, and producing a RunBuilder.

Constructor​

new EvalRunner(
  datasetSource: Record<string, unknown>,
  options?: {
    toolComparator?: ToolComparator;
    textComparator?: TextComparator;
    similarityThresholds?: { exact?: number; similar?: number };
  },
)

Parameters:

ParameterTypeRequiredDefaultDescription
datasetSourceRecord<string, unknown>Yes-The dataset_source object from a dataset (contains "runs")
options.toolComparatorToolComparatorNocompareToolArgsCustom (expected, actual) => [status, notes] function
options.textComparatorTextComparatorNotextSimilarityCustom (textA, textB) => number function
options.similarityThresholdsobjectNo{ exact: 0.70, similar: 0.40 }Score thresholds for match status

Type aliases:

type ToolComparator = (
  expected: Record<string, unknown>,
  actual: Record<string, unknown>,
) => [string, string | null];

type TextComparator = (a: string, b: string) => number;

The EvalRunner does not perform local grading. It pairs expected vs actual tool calls and text responses, then submits everything for server-side grading via the backend's LLM-based judge. Tool call arguments are compared structurally using compareArgsStructural(). Text responses are submitted with match_status="pending" for server-side evaluation.

Example:

import { EvalRunner } from "ashr-labs";

const runner = new EvalRunner(source);

// With custom thresholds
const runner = new EvalRunner(source, {
  similarityThresholds: { exact: 0.85, similar: 0.50 },
});

EvalRunner.fromDataset (static method)​

Create an EvalRunner by fetching a dataset from the API.

static async EvalRunner.fromDataset(
  client: AshrLabsClient,
  datasetId: number,
  options?: {
    toolComparator?: ToolComparator;
    textComparator?: TextComparator;
    similarityThresholds?: { exact?: number; similar?: number };
  },
): Promise<EvalRunner>

Parameters:

ParameterTypeRequiredDescription
clientAshrLabsClientYesAn authenticated client
datasetIdnumberYesThe dataset ID to fetch
optionsobjectNoPassed to EvalRunner constructor

Returns: EvalRunner — A configured runner ready to call .run()

Example:

const runner = await EvalRunner.fromDataset(client, 322);

EvalRunner.run​

Run the agent against every scenario and return a populated RunBuilder.

async runner.run(
  agent: Agent | (() => Agent),
  options?: {
    onScenario?: OnScenarioCallback;
    onAction?: OnActionCallback;
    onEnvironment?: OnEnvironmentCallback;
    maxWorkers?: number;
  },
): Promise<RunBuilder>

Parameters:

ParameterTypeRequiredDefaultDescription
agentAgent | (() => Agent)Yes-An object implementing the Agent interface, or a factory function
options.onScenarioOnScenarioCallbackNoundefinedCalled at the start of each scenario: (scenarioId, scenarioDict)
options.onActionOnActionCallbackNoundefinedCalled for each action: (actionIndex, actionDict)
options.onEnvironmentOnEnvironmentCallbackNoundefinedCalled for environment actions: `(content, actionDict) => object
options.maxWorkersnumberNo1Number of scenarios to run in parallel. When >1, scenarioId is passed to respond() and reset() so the agent can key state per scenario.

Type aliases:

type OnScenarioCallback = (scenarioId: string, scenario: Record<string, unknown>) => void;
type OnActionCallback = (actionIndex: number, action: Record<string, unknown>) => void;
type OnEnvironmentCallback = (content: string, action: Record<string, unknown>) => Record<string, unknown> | null | undefined;

Returns: RunBuilder — A populated builder ready for .build() or .deploy()

Example:

// Sequential (default)
const run = await runner.run(agent);
const result = run.build();
console.log(result.aggregate_metrics);

// With environment handler — feed external context to the agent
const run = await runner.run(agent, {
  onEnvironment: (content, action) => agent.respond(content),
});

// Parallel — run 4 scenarios at a time
const run = await runner.run(agent, { maxWorkers: 4 });

// With factory function
const run = await runner.run(() => new MyAgent(), { maxWorkers: 4 });

EvalRunner.runAndDeploy​

Run the eval and submit results in one call.

async runner.runAndDeploy(
  agent: Agent | (() => Agent),
  client: AshrLabsClient,
  datasetId?: number,
  options?: {
    onScenario?: OnScenarioCallback;
    onAction?: OnActionCallback;
    onEnvironment?: OnEnvironmentCallback;
    maxWorkers?: number;
    tenantId?: number;
    runnerId?: number;
    agentId?: number | null;
  },
): Promise<Record<string, unknown>>

Parameters:

ParameterTypeRequiredDefaultDescription
agentAgent | (() => Agent)Yes-An object implementing the Agent interface, or a factory function
clientAshrLabsClientYes-An authenticated client
datasetIdnumberNoundefinedThe dataset to submit against
options.onScenarioOnScenarioCallbackNoundefinedCallback per scenario
options.onActionOnActionCallbackNoundefinedCallback per action
options.onEnvironmentOnEnvironmentCallbackNoundefinedCallback for environment actions (see run())
options.maxWorkersnumberNo1Number of scenarios to run in parallel (default sequential)
options.tenantIdnumberNoautoThe tenant (auto-resolved if omitted)
options.runnerIdnumberNoundefinedID of the user who ran the test
options.agentIdnumberNonullAgent to auto-link the dataset to

Returns: The created run object from the API

Example:

// Sequential
const created = await runner.runAndDeploy(agent, client, 322);
console.log(`Run #${created.id} submitted`);

// Parallel
const created = await runner.runAndDeploy(agent, client, 322, { maxWorkers: 4 });

Agent Interface​

An interface that defines the contract agents must implement.

interface Agent {
  respond(
    message: string,
    scenarioId?: string,
  ): Record<string, unknown> | Promise<Record<string, unknown>>;

  reset(scenarioId?: string): void | Promise<void>;
}

respond​

Process a user message and return the agent's response.

Parameters:

ParameterTypeDescription
messagestringThe user's message text
scenarioIdstringOptional scenario ID (passed during parallel execution)

Returns: An object (or Promise of an object) with:

  • "text" (string): The agent's text response
  • "tool_calls" (Array): Tool calls made during this turn, each with "name" (string) and "arguments" (object) keys

arguments vs arguments_json: The Agent interface returns tool arguments as an object under the "arguments" key. However, RunBuilder and the API store them as a JSON string under "arguments_json". EvalRunner handles this conversion automatically. If you use RunBuilder directly, pass "arguments_json" (a JSON string) to addToolCall(). The extractToolArgs() helper accepts both formats, so comparators work either way.

reset​

Clear conversation state for a new scenario. Called before each scenario begins.

Parameters:

ParameterTypeDescription
scenarioIdstringOptional scenario ID (passed during parallel execution)

Tools​

tool() makes one definition the single source of truth for a tool — its schema, its provider renderings, and its executor — so you never declare the same tool twice (once for the live agent, once for dataset generation). See Testing Your Agent for how it fits the eval workflow.

tool​

function tool(def: ToolDef): Tool

interface ToolDef {
  name: string;
  description?: string;
  parameters?: JsonSchema;   // the JSON Schema for the arguments object
  fn: ToolExecutor;          // (args: Record<string, unknown>) => unknown | Promise<unknown>
}

TypeScript has no runtime type information, so you supply the parameters JSON Schema explicitly (unlike the Python @tool, which infers it from type hints).

const lookupOrder = tool({
  name: "lookup_order",
  description: "Look up a customer order.",
  parameters: {
    type: "object",
    properties: { order_id: { type: "string" } },
    required: ["order_id"],
  },
  fn: ({ order_id }) => ({ order_id, status: "shipped" }),
});

Tool​

new Tool(def: ToolDef)
MethodReturns
toAnthropic(){ name, description, input_schema } — Anthropic Messages API
toOpenAI(){ type: "function", function: { name, description, parameters } } — OpenAI
toAshr(){ name, description, parameters } — dataset generation
call(args)Invokes fn (it's the executor)

Readonly properties: name, description, parameters.

Agent adapters​

Drop-in Agent implementations that run the provider tool loop and accumulate tool_calls for grading, so you don't hand-roll it. If your agent is an Anthropic or OpenAI tool-calling loop, pass your tool() objects to one of these instead of writing an agent object. The Anthropic/OpenAI adapters need their provider SDK installed (@anthropic-ai/sdk / openai, optional peer deps). See Testing Your Agent → Step 1 for the end-to-end usage.

AnthropicAgent​

new AnthropicAgent(options: AnthropicAgentOptions)

interface AnthropicAgentOptions {
  model: string;
  system?: string;
  tools?: (Tool | Record<string, unknown>)[];
  client?: unknown;                       // your own Anthropic client; else constructed lazily
  execute?: Record<string, (args: Record<string, unknown>) => unknown>;
  maxTokens?: number;                     // default 1024
}

Runs the Anthropic tool loop (up to 10 iterations) and returns { text, tool_calls: { name, arguments }[] }. tools accepts Tool objects and/or raw Anthropic schema dicts; pass executors for raw dicts via execute. If client is omitted, one is constructed lazily and needs the @anthropic-ai/sdk package plus ANTHROPIC_API_KEY. reset() clears the conversation.

OpenAIAgent​

new OpenAIAgent(options: OpenAIAgentOptions)

interface OpenAIAgentOptions {
  model: string;
  system?: string;
  tools?: (Tool | Record<string, unknown>)[];
  client?: unknown;
  execute?: Record<string, (args: Record<string, unknown>) => unknown>;
}

Same contract for OpenAI Chat Completions. Lazy client needs the openai package and OPENAI_API_KEY.

FunctionAgent​

new FunctionAgent(
  fn: (message: string) => unknown | Promise<unknown>,
  options?: { resetFn?: () => void },
)

Wraps any callable that takes the user message and returns a string or a { text, tool_calls } object (sync or async). A bare-string return is normalized to { text: <string>, tool_calls: [] }; return the object form to get tool-call grading. reset() calls resetFn if provided.

Comparator Functions​

All comparator functions are standalone and importable from the top-level package.

stripMarkdown​

Remove markdown formatting from text.

stripMarkdown(text: string): string

Removes bold/italic markers, headers, bullets, and markdown links. Collapses whitespace.

Example:

stripMarkdown("**Bold** and [link](https://x.com)");
// => "Bold and link"

tokenize​

Lowercase, strip markdown and punctuation, split into word tokens.

tokenize(text: string): string[]

Example:

tokenize("Order **ORD-123** shipped!");
// => ["order", "ord123", "shipped"]

fuzzyStrMatch​

Check if two strings are semantically close enough to count as matching.

fuzzyStrMatch(a: string, b: string, threshold?: number): boolean

Parameters:

ParameterTypeRequiredDefaultDescription
astringYes-First string
bstringYes-Second string
thresholdnumberNoadaptiveWord-overlap threshold. If undefined: 0.35 for <=5 words, 0.40 for <=8, 0.55 otherwise

Returns: true if the strings match closely enough

Checks in order: exact match after normalization, containment, then word-set overlap.

Example:

fuzzyStrMatch("Customer wants a refund", "customer wants refund");  // true
fuzzyStrMatch("apple banana", "cherry grape");                      // false

extractToolArgs​

Extract arguments from a tool call object, handling both formats.

extractToolArgs(toolCall: Record<string, unknown>): Record<string, unknown>

Handles { arguments: {...} } (object form) and { arguments_json: "..." } (JSON string form). Prefers the object form if both are present.

Example:

extractToolArgs({ arguments_json: '{"order_id": "ORD-123"}' });
// => { order_id: "ORD-123" }

extractToolArgs({ arguments: { order_id: "ORD-123" } });
// => { order_id: "ORD-123" }

compareToolArgs​

Compare expected vs actual tool call arguments.

compareToolArgs(
  expected: Record<string, unknown>,
  actual: Record<string, unknown>,
): [string, string | null]

Parameters:

ParameterTypeDescription
expectedRecord<string, unknown>Expected tool call (with arguments or arguments_json)
actualRecord<string, unknown>Actual tool call made by the agent

Returns: A tuple of [matchStatus, divergenceNotes]:

  • matchStatus: "exact", "partial", or "mismatch"
  • divergenceNotes: Human-readable diff summary, or null if exact

String arguments are compared using fuzzyStrMatch. Non-string values use JSON.stringify equality. Extra arguments in the actual call don't cause divergence.

Example:

const [status, notes] = compareToolArgs(
  { arguments: { order_id: "ORD-123" } },
  { arguments: { order_id: "ORD-123", extra: "field" } },
);
// => ["exact", null]

const [status, notes] = compareToolArgs(
  { arguments: { order_id: "ORD-123", reason: "damaged item" } },
  { arguments: { order_id: "ORD-999", reason: "item was damaged" } },
);
// => ["partial", "'order_id': expected='ORD-123' actual='ORD-999'"]

textSimilarity​

Compute similarity between two text strings.

textSimilarity(textA: string, textB: string): number

Returns: A number between 0.0 and 1.0

Uses cosine similarity on word frequency vectors, plus:

  • Entity bonus (+0.20): for matching order IDs (ORD-*), refund IDs (REF-*), prices ($*), dates (YYYY-MM-DD), and tracking URLs
  • Concept bonus (+0.10): for matching domain concepts (refund/credited, shipped/transit/delivered, stock/available, etc.)

Example:

textSimilarity(
  "Your order ORD-123 has shipped and is on the way",
  "Order ORD-123 has been shipped and is in transit",
);
// => 0.78

Data Types​

User​

interface User {
  id?: number;
  created_at?: string;
  email?: string;
  name?: string | null;
  tenant?: number;
  is_active?: boolean;
}

Tenant​

interface Tenant {
  id?: number;
  created_at?: string;
  tenant_name?: string;
  is_active?: boolean;
}

Session​

interface Session {
  status: string;
  user: User;
  tenant: Tenant;
}

Dataset​

interface Dataset {
  id?: number;
  created_at?: string;
  tenant?: number;
  creator?: number;
  name?: string;
  description?: string | null;
  dataset_source?: Record<string, unknown>;
}

Run​

interface Run {
  id?: number;
  created_at?: string;
  dataset?: number;
  tenant?: number;
  runner?: number;
  result?: Record<string, unknown>;
}

ObservabilityTrace​

interface ObservabilityTrace {
  id?: string;
  name?: string;
  user_id?: string | null;       // End-user identifier
  session_id?: string | null;    // Conversation/session grouping
  metadata?: Record<string, unknown> | null;
  tags?: string[];
  created_at?: string | null;
  output?: unknown | null;
  observations?: ObservabilityObservation[];
}

ObservabilityObservation​

interface ObservabilityObservation {
  id?: string;
  name?: string;
  type?: string;                 // "span", "generation", "event"
  parent_observation_id?: string | null;
  input?: unknown | null;
  output?: unknown | null;
  metadata?: Record<string, unknown> | null;
  model?: string | null;         // LLM model name (generations only)
  usage?: { input_tokens?: number; output_tokens?: number } | null;
  level?: "DEBUG" | "DEFAULT" | "WARNING" | "ERROR" | null;
  status_message?: string | null;
  start_time?: string | null;
  end_time?: string | null;
}

SdkNote​

interface SdkNote {
  id?: number;
  created_at?: string;
  updated_at?: string;
  title?: string;
  content?: string;
  category?: string;   // "info" | "warning" | "breaking_change" | "best_practice" | "deprecation"
  severity?: string;   // "info" | "warning" | "critical"
  tenant_id?: number | null;
  agent_id?: number | null;
  active_from?: string;
  expires_at?: string | null;
  is_archived?: boolean;
  note_metadata?: Record<string, unknown>;
}

Request​

interface Request {
  id?: number;
  created_at?: string;
  requestor_id?: number;
  requestor_tenant?: number;
  request_name?: string;
  request_status?: string;
  request_input_schema?: Record<string, unknown> | null;
  request?: Record<string, unknown>;
}

APIKey​

interface APIKey {
  id?: number;
  key?: string;           // Only present on creation
  key_prefix?: string;
  name?: string;
  scopes?: string[];
  user_id?: number;
  tenant_id?: number;
  created_at?: string;
  last_used_at?: string | null;
  expires_at?: string | null;
  is_active?: boolean;
}

ToolCall​

interface ToolCall {
  name?: string;
  arguments_json?: string;
}

ExpectedResponse​

interface ExpectedResponse {
  tool_calls?: ToolCall[];
  text?: string;
}

Action​

interface Action {
  actor?: string;           // "user" or "agent"
  content?: string;
  name?: string;
  expected_response?: ExpectedResponse;
}

Scenario​

interface Scenario {
  title?: string;
  actions?: Action[];
}

Agent​

interface Agent {
  id: number;
  created_at: string;
  tenant_id: number;
  creator_id?: number | null;
  name: string;
  description?: string | null;
  config: AgentConfig;
  is_active: boolean;
  dataset_count: number;
}

AgentConfig​

interface AgentConfig {
  form_data?: Record<string, unknown>;   // Saved generation preset — the consumed field;
                                          // pre-fills new requests (tools at form_data.agent.tools)
  tool_definitions?: ToolDefinition[];    // optional descriptive metadata (not read by the platform)
  behavior_rules?: BehaviorRule[];        // optional descriptive metadata (not read by the platform)
}

ToolDefinition​

interface ToolDefinition {
  name: string;
  description?: string;
  required?: boolean;   // true = must be called, false = optional
}

BehaviorRule​

interface BehaviorRule {
  rule: string;
  strictness?: string;   // "required" | "expected" | "optional"
}