> ## Documentation Index
> Fetch the complete documentation index at: https://docs.skybridge.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Check what the model does with your app, in a test

A tool that works is not the same as a tool the model decides to call. [DevTools](https://docs.skybridge.tech/test/devtools.md) and the [playground](https://docs.skybridge.tech/test/playground.md) let you check that by hand; evals check it in a test. `@skybridge/test` runs a real conversation against your app, in process, and hands you the tool calls the model made so you can assert on them.

## Set Up

Add the test runtime, vitest, the AI SDK, and the provider you want to drive the conversation:

```bash npm theme={null}
npm install -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic
```

```bash pnpm theme={null}
pnpm add -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic
```

```bash yarn theme={null}
yarn add -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic
```

```bash bun theme={null}
bun add -d @skybridge/test vitest@^4 ai @ai-sdk/anthropic
```

Turn on `evals` in the Vite plugin. It registers the `expect.chat` matchers, picks up `evals/**/*.eval.ts`, raises the per-scenario timeout to two minutes, and loads your `.env` so the provider key is available:

```ts vite.config.ts highlight={7} theme={null}
import { skybridge } from "@skybridge/vite-plugin";
import react from "@vitejs/plugin-react";
import { defineConfig } from "vite";

export default defineConfig({
  plugins: [
    skybridge({ evals: {} }),
    react(),
  ],
});
```

Then add a script to run them:

```json package.json theme={null}
"evals": "vitest run evals"
```

## Write a Scenario

A scenario opens a conversation against your [app](https://docs.skybridge.tech/api-reference/skybridge.md), sends a prompt, and asserts on the tools the model called:

```ts evals/search.eval.ts theme={null}
import { anthropic } from "@ai-sdk/anthropic";
import { start } from "@skybridge/test";
import { expect, it } from "vitest";
import { app } from "../src/server.js";

it("reaches the search tool from a natural prompt", async () => {
  const chat = await start({ app, model: anthropic("claude-sonnet-4-5") });
  await chat.send("Find me running shoes under 100 dollars");

  expect.chat(chat).toHaveCalledToolWith("search-products", { query: "running shoes" });
});
```

`start` serves the app in process: no port, no fixture. Each `send` is one turn, during which the model can call several tools before it answers. The session closes when the test finishes.

This is why the app lives in `src/server.ts` and `run()` in `src/index.ts`: importing the app starts nothing.

## Assert on Tool Calls

`expect.chat(chat)` is typed against your app, so tool names autocomplete and arguments are checked against each tool's input schema.

| Matcher | Passes when |
| - | - |
| `toHaveCalledToolOnce(name, args?)` | exactly one successful call to `name`, optionally matching `args` |
| `toHaveCalledToolWith(name, args)` | some successful call to `name` matched `args` |
| `toNeverHaveCalledTool(name)` | `name` was never attempted |
| `toHaveFailedToolCall(name)` | a call to `name` was refused or threw |
| `toHaveSaid(text)` | an assistant turn contains `text` (string or `RegExp`) |
| `toHaveCalledToolsInOrder(...names)` | the named tools were called in that relative order |
| `toHaveCalledNoTools()` | no tool was attempted at all |
| `toPassJudgment(criteria)` | a judge model grades the conversation against `criteria` |

`toHaveCalledToolsInOrder` matches a subsequence, so unrelated calls in between do not break it. Like the other positive matchers it only counts calls that succeeded, so a failed call never satisfies a step.

Every matcher supports `.not` and sees the whole conversation, not just the last `send`. On failure, the message lists the calls the model actually made, arguments included. `chat.toolCalls` and `chat.assistantTurns` are also available for custom assertions.

## Judge the Answer

Sometimes it's not easy to test an agent behavior using on deterministic matchers. For example, did the assistant reply using the right tone? Was it polite? `toPassJudgment` sends the conversation to model, asking it to judge if the conversation meets the criteria:

```ts evals/trip.eval.ts theme={null}
it("plans a trip within budget", async () => {
  const chat = await start({ app, model: anthropic("claude-sonnet-4-5") });
  await chat.send("Plan a weekend in Lisbon under 500 euros");

  await expect.chat(chat).toPassJudgment(
    "proposes a concrete plan, stays under the budget, and relies on the tool result rather than inventing prices",
  );
});
```

The reasoning is the failure message:

```
expected the conversation to pass judgment:
  "proposes a concrete plan, stays under the budget, and relies on the tool result rather than inventing prices"

judge: FAIL
  The plan totals 640 euros. The assistant quoted hotel prices that do not appear in the search-hotels result.
```

By default, the judge runs on the chat's own model. Pass `model` to use another one:

```ts theme={null}
await expect.chat(chat).toPassJudgment("answers in one short paragraph", {
  model: anthropic("claude-haiku-4-5"),
});
```

### Judge With Something Other Than a Model

Pass `judge` to grade the conversation yourself. It receives the criteria and the rendered transcript, and returns a verdict. This is how you plug in an evaluation model, which answers a typed question with a probability instead of writing text, or any scoring service you already run:

Define it once in its own module, so every scenario that wants it imports the same judge:

```ts evals/typesafe-judge.ts theme={null}
import type { Judgment, Verdict } from "@skybridge/test";
import { TypeSafeClient, noul } from "@typesafe-ai/sdk";

export async function typesafeJudge({
  criteria,
  transcript,
}: Judgment): Promise<Verdict> {
  const { answers } = await new TypeSafeClient().systemOne({
    state: { conversation: transcript },
    questions: { pass: noul(criteria) },
  });

  return {
    pass: answers.pass.noul >= 0.5,
    reasoning: `P(pass) = ${answers.pass.noul.toFixed(2)}`,
  };
}
```

```ts evals/trip.eval.ts theme={null}
import { typesafeJudge } from "./typesafe-judge.js";

await expect
  .chat(chat)
  .toPassJudgment("stays under the budget", { judge: typesafeJudge });
```

`Judgment` and `Verdict` are exported from `@skybridge/test`, so a judge written this way is typed without repeating the shape.

`reasoning` is optional, and the failure message drops the line when there is none. A judge that returns no prose is cheaper and faster, but a red eval then tells you only that it failed, so it is worth returning the score.

By default a judgment is a live model call: it costs money, and it is not reproducible the way a tool-call assertion is. A custom `judge` costs and varies only as much as whatever it calls, and a local heuristic is free and deterministic. Either way, use it only when no other matcher fits, and keep the criteria narrow so a failure points at one thing.

When the judge throws, the assertion throws `judge unavailable: <error>` instead of failing. That covers a broken provider and a bug in your own callback alike, so neither reads as an app regression.

## Stub a Tool Result

A scenario that depends on today's date or on a live catalogue passes today and fails next month, with no change to your code. Pass `stubs` to answer a tool from the scenario instead of the app:

```ts evals/flights.eval.ts highlight={4-6} theme={null}
const chat = await start({
  app,
  model: anthropic("claude-sonnet-4-5"),
  stubs: {
    "search-flights": ({ from, to }) => ({ flights: [{ from, to, price: 120, departsAt: "2026-04-03T08:15:00Z" }] }),
  },
});
```

The arguments are typed against your registry, and a stubbed call still lands in `chat.toolCalls`, so every matcher works on it. Return a string or any JSON value: the runner wraps it the way a real tool result reaches the model. Naming a tool your app does not expose fails the scenario rather than passing silently. Return `undefined` to fall through to the real server, which is how you pin one set of arguments and leave the rest live:

```ts theme={null}
stubs: {
  "search-flights": ({ to }) => (to === "LIS" ? lisbonFixture : undefined),
}
```

A stub replaces the call: the tool never reaches your handler, so neither its schema validation nor its scope checks run. Stub the tools whose results move under you, and leave the ones the scenario is actually about.

## Test Authenticated Tools

For an app behind [OAuth](https://docs.skybridge.tech/build/auth.md), claim an identity for the session:

```ts evals/checkout.eval.ts highlight={4} theme={null}
const chat = await start({
  app,
  model: anthropic("claude-sonnet-4-5"),
  authInfo: { token: "eval", clientId: "evals", scopes: ["checkout"], extra: { email: "ada@example.com" } },
});
```

Only token verification is skipped. Per-tool schemes and scope checks run for real against those claims, and `extra` reaches your handlers as `extra.http.authInfo.extra`. Omit `authInfo` to exercise the anonymous path, auth challenges included.

> [!NOTE]
>
> `setup` and `oauth` still resolve on the first request, so an app wired to an identity provider needs its `.env` to run evals, the same as `dev`.

## Tune the Run

Defaults every scenario starts from go in the plugin option, and any of them can be overridden per `start`:

```ts vite.config.ts theme={null}
skybridge({
  evals: {
    temperature: 0,          // unset by default: the model's own default applies
    systemPrompt: "You are a shopping assistant.",
    maxSteps: 8,             // tool-call rounds per turn, default
    timeout: 120_000,        // per scenario, default
  },
})
```

Evals are live model calls: they cost money and the wording varies from run to run. Set the temperature to `0` when your model supports it (some recent models reject the parameter), assert on the calls rather than on exact phrasing, and reach for `toHaveSaid` with a loose pattern when the answer itself matters.

## Go Further

> [card] [Skybridge](https://docs.skybridge.tech/api-reference/skybridge.md)
>
> The app your scenarios import

> [card] [Playground](https://docs.skybridge.tech/test/playground.md)
>
> Chat with a real model running your app

> [card] [Register Tools](https://docs.skybridge.tech/build/tools.md)
>
> Write descriptions the model can act on

## Attach Input Files

`chat.send` accepts AI SDK `UserContent`: either a string or an array of text and file parts. Use the AI SDK's `FilePart` directly, and keep that same object for assertions:

```ts evals/resize.eval.ts theme={null}
import { readFile } from "node:fs/promises";
import type { FilePart } from "ai";

const portrait: FilePart = {
  type: "file",
  mediaType: "image/png",
  filename: "photo.png",
  data: await readFile(new URL("./fixtures/photo.png", import.meta.url)),
};

await chat.send([
  { type: "text", text: "Resize this photo to a vertical 1080x1920 portrait." },
  portrait,
]);

expect.chat(chat).toHaveCalledToolWith("image_crop_and_resize", {
  file: expect.inputFile(portrait),
});

expect.chat(chat).toHaveCalledToolOnce("asset_openai_file_upload", {
  files: [expect.inputFile(portrait)],
});
```

Declare file parameters with `FileRef` and list their names in `_meta["openai/fileParams"]`. Your handler or stub receives the attached file, and its download URL remains available until the conversation closes.

`expect.inputFile(part)` matches the file object minted for that exact `FilePart`. Two parts with identical bytes do not match each other.

Using it under a parameter not listed in `openai/fileParams` throws a declaration error before calls are compared, including nested matchers, `.not` assertions, and conversations with no calls.

File data can be bytes, base64, a URL, or AI SDK tagged data, URL or text content. Provider file references are rejected because the app under test cannot download them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.