> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trajectory.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate and compare

> Score a model or checkpoint on a spec (benchmark).

An evaluation runs a [spec](https://docs.trajectory.ai/guides/spec.md) and reads the grader's rewards.
Use `bench_id` to select an exact spec version. `agent_name` with
`benchmark_name` selects that agent's latest version; `agent_id` identifies ownership
but does not select a spec by itself.

## 1. Start a baseline

```python theme={null}
from trajectory import Client

client = Client()
bench_id = "YOUR_BENCHMARK_ID"

for model in client.evals.list_options(bench_id=bench_id).models:
    print(model.base_model_slug, model.options)

run = client.evals.create(
    bench_id=bench_id,
    base_model_slug="YOUR_SUPPORTED_MODEL_SLUG",
    options={"val_num_tasks": 16},
    display_name="Baseline",
)
print(run.eval_run_id)
```

An evaluation accepts these `options` ([training options](https://docs.trajectory.ai/guides/training.md#training-options)
describes each): `val_num_tasks`,
`val_trajectories_per_task`, `val_max_concurrent_trajectories`,
`val_final_wait_timeout_seconds`, `reasoning_effort`, `temperature`, `top_p`,
`top_k`, `trajectory_timeout_seconds`, `max_output_tokens_per_turn`,
`max_output_tokens_per_trajectory`, and `max_turns_per_trajectory`. Other options return
422\. Use `list_options` to read evaluation defaults, or omit `options` to use them.
Training and standalone evaluation both default to 200 turns per trajectory.

An evaluation runs the spec's `test` tasks; pass `task_split="all"` to include its
`train` tasks. Omit `val_num_tasks` to run every selected task. To evaluate a
ready deployment owned by the spec's agent, pass `deployment_id=` instead of
`base_model_slug`; it uses the deployment's sampling, token, and turn settings, so those
options return 422.

After task diagnostics, run a small evaluation to check the managed model, grader,
and published reward before a larger held-out baseline.

## 2. Read the score

```python theme={null}
progress = client.evals.runs.progress(run.eval_run_id)
print(progress.status, progress.terminal_rollouts, progress.total_rollouts)

result = client.evals.runs.retrieve(run.eval_run_id)
print(result.status, result.reward_mean)

for item in client.evals.rewards.list_trajectory_rewards(run.eval_run_id):
    print(item.task_id, item.reward)
```

Poll progress until `status` is `completed`; failed or cancelled runs are terminal too.
Read the mean reward after completion. A failed run can contain graded trajectories without
a published mean. Count attempted, graded, failed, and unfinished tasks before comparing
means, and retrieve a trajectory with `include_reward_components=True` to check its native
score, ending, and failure. `client.evals.runs.list(bench_id)` lists the spec's evaluations.
Open the agent's **Evaluation** tab for the same scores. See [Use the platform](https://docs.trajectory.ai/platform.md)
for direct run links.

To read what the model said, take a trajectory ID from those rewards and read its steps.
Each step holds the conversation so far, so the last step is the full transcript:

```python theme={null}
tid = item.trajectory_id
num_steps = client.trajectories.retrieve(tid).num_steps
if num_steps:  # A trajectory that failed before its first model call has no steps.
    last = client.trajectories.steps.list(tid, start_step_index=num_steps - 1, limit=1).items[0]
    for message in last.messages or []:
        print(message.role, message.content)

# Every step, for per-step rewards; iterating fetches each following page.
for step in client.trajectories.steps.list(tid):
    print(step.step_index, step.reward)
```

`client.trajectories.retrieve(tid, include_steps=True)` returns every step in one response;
for long trajectories, page with `steps.list`.

## 3. Score a checkpoint

```python theme={null}
checkpoint = max(
    client.training.checkpoints.list("YOUR_TRAINING_RUN_ID"),
    key=lambda c: c.step_index,
)
checkpoint_eval = client.evals.create(
    bench_id=bench_id,
    base_model_slug=checkpoint.base_model_slug,
    parent_checkpoint_id=checkpoint.checkpoint_id,
    options={"val_num_tasks": 16},
    display_name="Checkpoint comparison",
)
print(checkpoint_eval.eval_run_id)
```

Compare against the same spec version, held-out task IDs, grader, and effective
generation limits. A checkpoint evaluation inherits the training run's resolved validation sampling,
token, turn, and timeout settings, including its `val_*` overrides. Explicit evaluation
`options` take precedence. A base-model evaluation uses evaluation defaults, so pass
the same explicit `options` to runs you compare. A capped evaluation on a new spec
version may select different tasks. Report per-task scores and coverage alongside the
mean. Then [deploy](https://docs.trajectory.ai/guides/deployments.md) a checkpoint that meets your criteria.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.