Evaluate and compare

An evaluation runs a spec and reads the grader’s rewards. Use bench_id to select an exact spec version. agent_name with benchmark_name selects that agent’s latest version; agent_id identifies ownership but does not select a spec by itself.

1. Start a baseline

from trajectory import Client

client = Client()
bench_id = "YOUR_BENCHMARK_ID"

for model in client.evals.list_options(bench_id=bench_id).models:
    print(model.base_model_slug, model.options)

run = client.evals.create(
    bench_id=bench_id,
    base_model_slug="YOUR_SUPPORTED_MODEL_SLUG",
    options={"val_num_tasks": 16},
    display_name="Baseline",
)
print(run.eval_run_id)

An evaluation accepts these options (training options describes each): val_num_tasks, val_trajectories_per_task, val_max_concurrent_trajectories, val_final_wait_timeout_seconds, reasoning_effort, temperature, top_p, top_k, trajectory_timeout_seconds, max_output_tokens_per_turn, max_output_tokens_per_trajectory, and max_turns_per_trajectory. Other options return 422. Use list_options to read evaluation defaults, or omit options to use them. Training and standalone evaluation both default to 200 turns per trajectory. An evaluation runs the spec’s test tasks; pass task_split="all" to include its train tasks. Omit val_num_tasks to run every selected task. To evaluate a ready deployment owned by the spec’s agent, pass deployment_id= instead of base_model_slug; it uses the deployment’s sampling, token, and turn settings, so those options return 422. After task diagnostics, run a small evaluation to check the managed model, grader, and published reward before a larger held-out baseline.

2. Read the score

progress = client.evals.runs.progress(run.eval_run_id)
print(progress.status, progress.terminal_rollouts, progress.total_rollouts)

result = client.evals.runs.retrieve(run.eval_run_id)
print(result.status, result.reward_mean)

for item in client.evals.rewards.list_trajectory_rewards(run.eval_run_id):
    print(item.task_id, item.reward)

Poll progress until status is completed; failed or cancelled runs are terminal too. Read the mean reward after completion. A failed run can contain graded trajectories without a published mean. Count attempted, graded, failed, and unfinished tasks before comparing means, and retrieve a trajectory with include_reward_components=True to check its native score, ending, and failure. client.evals.runs.list(bench_id) lists the spec’s evaluations. Open the agent’s Evaluation tab for the same scores. See Use the platform for direct run links. To read what the model said, take a trajectory ID from those rewards and read its steps. Each step holds the conversation so far, so the last step is the full transcript:

tid = item.trajectory_id
num_steps = client.trajectories.retrieve(tid).num_steps
if num_steps:  # A trajectory that failed before its first model call has no steps.
    last = client.trajectories.steps.list(tid, start_step_index=num_steps - 1, limit=1).items[0]
    for message in last.messages or []:
        print(message.role, message.content)

# Every step, for per-step rewards; iterating fetches each following page.
for step in client.trajectories.steps.list(tid):
    print(step.step_index, step.reward)

client.trajectories.retrieve(tid, include_steps=True) returns every step in one response; for long trajectories, page with steps.list.

3. Score a checkpoint

checkpoint = max(
    client.training.checkpoints.list("YOUR_TRAINING_RUN_ID"),
    key=lambda c: c.step_index,
)
checkpoint_eval = client.evals.create(
    bench_id=bench_id,
    base_model_slug=checkpoint.base_model_slug,
    parent_checkpoint_id=checkpoint.checkpoint_id,
    options={"val_num_tasks": 16},
    display_name="Checkpoint comparison",
)
print(checkpoint_eval.eval_run_id)

Compare against the same spec version, held-out task IDs, grader, and effective generation limits. A checkpoint evaluation inherits the training run’s resolved validation sampling, token, turn, and timeout settings, including its val_* overrides. Explicit evaluation options take precedence. A base-model evaluation uses evaluation defaults, so pass the same explicit options to runs you compare. A capped evaluation on a new spec version may select different tasks. Report per-task scores and coverage alongside the mean. Then deploy a checkpoint that meets your criteria.

Get started

Build and improve

Production learning

Reference

Evaluate and compare

1. Start a baseline

2. Read the score

3. Score a checkpoint

​1. Start a baseline

​2. Read the score

​3. Score a checkpoint

1. Start a baseline

2. Read the score

3. Score a checkpoint