Bring your spec (benchmark)

A spec contains tasks for the model to solve. Each task uses an image containing its dependencies, files and your harness. The harness reads the instructions, runs the model, grades its work and reports the result to Trajectory through the Python SDK. Each task specifies the command that starts the harness and any inputs or settings it needs. Multiple tasks can share an image, with different inputs for each task. For quick debugging, diagnose a representative task, then upload a small spec and evaluate it to confirm a recorded grade. When ready to expand, upload the full spec and diagnose a subset of its tasks before training. The cookbook has worked integrations. You’ll write two Python scripts:

harness.py runs inside the image. It reads the task instructions, calls the model, grades its work and reports the result.
upload.py runs on your computer. It defines the tasks and uploads the spec and its image files.

The examples below show the relevant pieces of each script. The “Update your spec” section is optional; use it when you want to update an existing spec. Before running the examples, choose or create an agent and use its name wherever YOUR_AGENT_NAME appears. Generally, agents should have a clear grouping of similar specs and tasks.

Define your tasks

Use BenchmarkSpec to describe the spec and TaskSpec for each task. Each task needs a unique name, a runtime (its own or the spec’s shared runtime), and a run_command that starts your harness inside that runtime. split is optional; tasks without one are assigned by a seeded shuffle, about 15% to test and the rest to train. If needed, you can pass task-specific inputs through env_vars:

upload.py

from trajectory import TaskSpec

task = TaskSpec(
    name="repair-config",
    split="test",
    run_command="python /app/harness.py",
    env_vars={"TASK_INSTRUCTIONS": "Fix the invalid configuration in /workspace/config.json."},
)

Your harness can read these environment variables with os.environ. For larger inputs, package the files in the image and have your harness read them. If tasks share an image, use task-specific settings—such as environment variables or command-line arguments in run_command—to select the right files and execute the task. Task variable names cannot start with TRAJECTORY_ or be MODEL_ENDPOINT_ID, MODEL_ENDPOINT_ACCESS_TOKEN, or MODEL_ENDPOINT_URL. Use SecretRef for credentials; managed tasks receive their Trajectory session key and base URL automatically. Set CPU and memory through env_resources. Task sandboxes have no network access by default. For source calls to known hosts, set env_resources.network_mode="allowlist" and list them in env_resources.allowed_hosts. Keep training and test tasks in their respective splits, and retain the source task IDs when selecting a subset. If a task runs Docker containers, set env_resources.docker_engine=True. The execution environment must be able to start a Docker daemon and pass docker info; installing only the Docker CLI is not enough.

Call the model using Trajectory Client

In managed runs, Trajectory provides credentials and the API URL, so your harness can use Client() without configuring them. Create the trajectory before task setup and keep its ID for the whole attempt. Pass it with each model request:

harness.py

import os

from trajectory import Client

client = Client()
tid = client.trajectories.create().tid
response = client.chat.completions.create(
    model="policy",
    messages=[{"role": "user", "content": os.environ["TASK_INSTRUCTIONS"]}],
    x_trajectory_id=tid,
)

In a managed evaluation or training run, the run selects the actor model; changing the request’s model does not select a different actor. Generation settings come from the run’s options, set when you start an evaluation or training run: max_output_tokens_per_turn replaces the request’s max_tokens, and the run’s temperature, top_p, top_k, and reasoning_effort take precedence over request values. Your harness decides whether one response is enough or whether the model needs more turns. When using Trajectory for a source-fixed judge or other auxiliary model, use an organization-key client and a separate trajectory. trajectories.create() with the managed session key reuses the actor’s trajectory. Pass the separate trajectory’s ID with each auxiliary model request: x_trajectory_id in the Trajectory SDK or X-Trajectory-Id in direct HTTP calls. Keep the organization key out of actor-visible tools. You can give the model tools if your harness supports executing tool calls and returning their results to the model. Each tool describes an action the model can request, such as running a shell command or reading a file. Include any software those tools need in the image and provide the harness code that executes them.

Report the grade through the SDK

Your verifier decides how to grade the model’s work. It might compare an answer with an expected value, run a test suite or inspect files and running services. For an existing spec, trace the score field or formula its evaluation reader uses. Keep the native scale and grading rule, including partial credit when that is the reported score; a successful grader command alone does not identify the score. Once your verifier produces a valid numeric grade, report it against the same trajectory ID, then mark the attempt complete:

harness.py

client.trajectories.log_reward(
    tid,
    name="reward_accuracy",
    value=grade,
)
client.trajectories.complete(tid)

Rewards default to [0, 1]. For a different scale, declare its bounds when logging:

harness.py

client.trajectories.log_reward(
    tid,
    name="score",
    value=-0.5,
    min_value=-1,
    max_value=1,
)

The effective reward is the weighted average of components scaled to [0, 1] by their declared bounds. Use weight=0 for a secondary score you want to inspect without changing that reward. Retrieve components with include_reward_components=True; trajectory.reward alone can be zero without a native grade. The source-grader guide shows a worked mapping and grader-output event. These snippets show the SDK calls to place in your harness; they are not a complete program. complete() defaults to ENV_DONE, which means the attempt finished normally. When it did not, pass the reason. For an unrecovered model, environment, or verifier failure, report ERROR even if an incidental grader output exists. Do not publish that output as reward or substitute zero:

harness.py

client.trajectories.complete(tid, termination_reason="ERROR")

Termination reasons explains reason selection, including cut-off replies and how endings affect training and evaluation. A completed attempt can have an incorrect answer. An error means the harness could not produce a valid grade. Upload any outputs and logs you need before completion, because the sandbox may be cleaned up afterward.

Package the harness in an image

Include your harness, task files and dependencies in the image. This Dockerfile installs the latest SDK and copies your application files into /app:

Dockerfile

FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir --upgrade trajectory-sdk
COPY . /app/

Here, run_command="python /app/harness.py" assumes harness.py is in that build context. Check that paths in your task instructions and commands match where the Dockerfile copies files. For the /workspace/... paths above, you could put the task files under benchmark/workspace/ locally and add COPY workspace/ /workspace/ to the Dockerfile. COPY . /app/ alone would put them under /app/workspace/ instead. A build context holds at most 4,096 files. For a larger task set, archive the task files, COPY the archive, and unpack it in a RUN step. ADD accepts only a single http:// or https:// URL, not local files or archives. Add the tools and verifier dependencies your task needs, keeping held-out tests out of any environment the model can inspect. Keep credentials out of the image. Test that separation through the actor’s actual file and shell tools; hiding a path from its prompt does not make the path private. DockerfileBuild uploads files under the Dockerfile’s directory. Scope that context to at most 4,096 files and 3 GiB; use .dockerignore or a smaller runtime directory when the checkout is larger. COPY paths are relative to that context. For harnesses that run Docker containers, such as Harbor, install and verify the Docker daemon, CLI and Compose during the image build. On Debian Trixie:

Dockerfile

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
       docker.io docker-cli docker-compose \
    && docker --version \
    && docker compose version \
    && dockerd --version \
    && rm -rf /var/lib/apt/lists/*

Validate one task before uploading

For quick debugging, run start_task_diagnostic() on one representative task. A diagnostic reports each task’s status and, for a failed task, its failure: a code, plus the rollout and trajectory IDs, termination reason, and a redacted summary of any harness exception when the task got that far:

pip install --upgrade trajectory-sdk
export TRAJECTORY_API_KEY="YOUR_TRAJECTORY_API_KEY"

import time
from pathlib import Path

from trajectory import Client, TaskSpec
from trajectory.lib import DockerfileBuild, start_task_diagnostic

client = Client()
task = TaskSpec(
    name="repair-config",
    runtime=DockerfileBuild("Dockerfile"),
    run_command="python /app/harness.py",
    env_vars={"TASK_INSTRUCTIONS": "Fix the invalid configuration in /workspace/config.json."},
)
diagnostic = start_task_diagnostic(
    client,
    task,
    agent_name="YOUR_AGENT_NAME",
    root=Path("benchmark"),
    options={"max_output_tokens_per_turn": 4096},
)
while True:
    status = client.diagnostics.get_status(diagnostic.benchmark_diagnostic_id)
    if status.status in {"completed", "failed", "cancelled"}:
        break
    time.sleep(5)
result = client.diagnostics.get_diagnostics(diagnostic.benchmark_diagnostic_id)
print(result.status, result.task_counts, result.failure)
for task_result in result.tasks:
    print(
        task_result.task_id,
        task_result.status,
        task_result.termination_reason,
        task_result.reward,
        task_result.failure,
    )

The helper handles uploads and runtime builds without an existing spec ID. Its timeout_seconds bounds only the image-build wait, not task execution. ImageRef(...) and RuntimeRef(...) also work if you already have an image or runtime. Check each task’s failure and the run-wide result.failure before moving on. A diagnostic makes one attempt per task on the API’s default diagnostic model, reported as result.base_model_slug; pass base_model_slug to choose another. options takes the settings an evaluation accepts, except the sample counts, which the diagnostic sets. Pass the model and settings you will evaluate and train with, so the diagnostic generates replies the same way. Each task reports its trajectory_id, the termination_reason it ended with, and its reward, which is null when the trajectory ended before the harness recorded one.

Upload the spec and build its image

Put your Dockerfile, harness and task files in a local benchmark/ directory. The example below uploads the task defined above to your chosen agent:

upload.py

from pathlib import Path

from trajectory import BenchmarkSpec, Client
from trajectory.lib import DockerfileBuild, push

client = Client()
benchmark = BenchmarkSpec(
    name="Configuration repair",
    runtime=DockerfileBuild("Dockerfile"),
    tasks=[task],
)
uploaded = push(
    client, benchmark, agent_name="YOUR_AGENT_NAME", root=Path("benchmark"), build_images=True
)
print(uploaded.bench_id)

This uploads your tasks and files and waits for the image build. If registration or building fails, push() raises BenchmarkOperationError, which you can import from trajectory.lib; inspect its failures for the affected runtime and build error. For an upload that may outlive your client, use start_push() and save its returned operation id. A local timeout does not stop ingestion: reconnect with get_operation(client, operation_id), then inspect refresh() and pages("failure") before retrying. A failed runtime blocks evaluation of that spec version; correct it and upload a new version rather than creating a second, partial spec. To use this reconnectable form instead of the push() call above:

from trajectory.lib import start_push

operation = start_push(
    client, benchmark, agent_name="YOUR_AGENT_NAME", root=Path("benchmark"), build_images=True
)
print(operation.id)
result = operation.result(timeout=45 * 60)
print(result.status.bench_id)

Uploading does not run the tasks. After the diagnostic passes, run a small evaluation and confirm a recorded grade. A graded zero is valid; use held-out tasks when comparing checkpoints. push(), start_push(), and start_task_diagnostic() accept upload_timeout in seconds for blob store uploads, defaulting to 3600 (one hour). Pass upload_timeout=None to disable blob store timeouts. These apply to individual HTTP operations against the blob store, separately from API request timeouts, image-build waits, and signed-URL expiry. They do not set a total upload deadline. push() and start_push() also accept upload_max_retries, the retries for each upload request (default 32, within 4.5 minutes of waiting). See Errors and retries. You can share a runtime across tasks or set runtime on individual tasks:

Runtime definition	What you provide
`DockerfileBuild("Dockerfile")`	A Dockerfile and the files in its directory.
`ImageRef("registry.example.com/agent@sha256:...")`	An existing image pinned to a digest.
`RuntimeRef("YOUR_RUNTIME_ID")`	A runtime already registered in your organization.

Import these runtime helpers from trajectory.lib. For secret task inputs, use SecretRef(secret_ref="SECRET_NAME") in env_vars instead of packaging secret values.

Upload and diagnose the full spec

Expand BenchmarkSpec.tasks to the full task set and upload it with push(). Once push() finishes, call client.diagnostics.start_benchmark() using that upload’s bench_id. It runs up to 10 tasks, one attempt each, starting with one task per runtime, and reports the same task outcomes and failures as the single-task diagnostic. You need not repeat the single-task diagnostic for an unchanged runtime.

import time

diagnostic = client.diagnostics.start_benchmark(
    bench_id=uploaded.bench_id, options={"max_output_tokens_per_turn": 4096}
)
while True:
    status = client.diagnostics.get_status(diagnostic.benchmark_diagnostic_id)
    if status.status in {"completed", "failed", "cancelled"}:
        break
    time.sleep(5)

report = client.diagnostics.get_diagnostics(diagnostic.benchmark_diagnostic_id)
print(report.status, report.task_counts, report.failure)
for task_result in report.tasks:
    print(
        task_result.task_id,
        task_result.status,
        task_result.termination_reason,
        task_result.reward,
        task_result.failure,
    )

Inspect the run-wide failure and every task’s outcome, ending, and reward. If the diagnostic fails, fix the reported problem, upload the revised full spec, and diagnose that version. After it passes, run an evaluation to confirm a recorded reward before training. Evaluation and training accept agent_name and benchmark_name together to select the latest version by name. Use bench_id when you need a specific version; spec diagnostics require the ID returned by the upload helper.

Update your spec

To upload a revised spec, use the same spec name and agent and include the full set of tasks you want in the new version. Earlier versions remain available. Evaluation and training by name select the latest version, so finish a baseline/checkpoint comparison before uploading another version under the same name.

Tips for adapting existing specs

Your harness implements the deadlines, grading environments and evidence uploads below.

Bring the verifier as well as the task environment. Check that its scripts, dependencies and expected paths are available when grading starts. A ready image does not prove the verifier can run; test one task before launching the full set.
Give the agent and verifier separate time limits. When the agent runs out of time, stop its model calls and tools, then grade the work completed so far. Leave the verifier its full allowance and account for both phases in the rollout timeout.
Follow the spec’s grading rules. Preserve resource limits, startup commands and environment settings. Some specs grade in the same running container; others require applying the model’s changes to a fresh one. If the model has shell or file access, your harness must keep held-out tests outside the environment those tools can inspect. Uploading an image does not automatically provide this separation.
Keep enough evidence to explain a result. Save the generated output, verifier logs and error details as artifacts before completing the trajectory. Report missing logs separately from missing grades, and state any changes to the original spec.

Get started

Build and improve

Production learning

Reference

Bring your spec (benchmark)

Define your tasks

Call the model using Trajectory Client

Report the grade through the SDK

Package the harness in an image

Validate one task before uploading

Upload the spec and build its image

Upload and diagnose the full spec

Update your spec

Tips for adapting existing specs

​Define your tasks

​Call the model using Trajectory Client

​Report the grade through the SDK

​Package the harness in an image

​Validate one task before uploading

​Upload the spec and build its image

​Upload and diagnose the full spec

​Update your spec

​Tips for adapting existing specs

Define your tasks

Call the model using Trajectory Client

Report the grade through the SDK

Package the harness in an image

Validate one task before uploading

Upload the spec and build its image

Upload and diagnose the full spec

Update your spec

Tips for adapting existing specs