> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trajectory.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring your spec (benchmark)

> Bring your own tasks, agent harness, and grading.

A spec contains tasks for the model to solve. Each task uses an image containing
its dependencies, files and your harness. The harness reads the instructions, runs the model,
grades its work and reports the result to Trajectory through the Python SDK.

Each task specifies the command that starts the harness and any inputs or settings it
needs. Multiple tasks can share an image, with different inputs for each task.

For quick debugging, diagnose a representative task, then upload a small spec and
[evaluate](https://docs.trajectory.ai/guides/evaluations.md) it to confirm a recorded grade. When ready to expand,
upload the full spec and diagnose a subset of its tasks before training. The
[cookbook](https://github.com/Trajectorylabs/trajectory-cookbook) has worked integrations.

You'll write two Python scripts:

* **`harness.py`** runs inside the image. It reads the task instructions, calls the
  model, grades its work and reports the result.
* **`upload.py`** runs on your computer. It defines the tasks and uploads the spec
  and its image files.

The examples below show the relevant pieces of each script. The "Update your spec"
section is optional; use it when you want to update an existing spec.

Before running the examples, [choose or create an agent](https://docs.trajectory.ai/agent.md) and use its name wherever
`YOUR_AGENT_NAME` appears. Generally, agents should have a clear grouping of similar specs
and tasks.

## Define your tasks

Use `BenchmarkSpec` to describe the spec and `TaskSpec` for each task. Each task
needs a unique name, a runtime (its own or the spec's shared runtime), and a
`run_command` that starts your harness inside that runtime. `split` is optional; tasks
without one are assigned by a seeded shuffle, about 15% to `test` and the rest to `train`.

If needed, you can pass task-specific inputs through `env_vars`:

```python upload.py theme={null}
from trajectory import TaskSpec

task = TaskSpec(
    name="repair-config",
    split="test",
    run_command="python /app/harness.py",
    env_vars={"TASK_INSTRUCTIONS": "Fix the invalid configuration in /workspace/config.json."},
)
```

Your harness can read these environment variables with `os.environ`. For larger inputs,
package the files in the image and have your harness read them. If tasks share an image,
use task-specific settings—such as environment variables or command-line arguments in
`run_command`—to select the right files and execute the task.
Task variable names cannot start with `TRAJECTORY_` or be `MODEL_ENDPOINT_ID`,
`MODEL_ENDPOINT_ACCESS_TOKEN`, or `MODEL_ENDPOINT_URL`. Use `SecretRef` for credentials;
managed tasks receive their Trajectory session key and base URL automatically.

Set CPU and memory through `env_resources`. Task sandboxes have no network access by
default. For source calls to known hosts, set `env_resources.network_mode="allowlist"`
and list them in `env_resources.allowed_hosts`. Keep training and test tasks in their
respective splits, and retain the source task IDs when selecting a subset.

If a task runs Docker containers, set `env_resources.docker_engine=True`. The execution
environment must be able to start a Docker daemon and pass `docker info`; installing
only the Docker CLI is not enough.

## Call the model using Trajectory Client

In managed runs, Trajectory provides credentials and the API URL, so your harness can
use `Client()` without configuring them.

Create the trajectory before task setup and keep its ID for the whole attempt.
Pass it with each model request:

```python harness.py theme={null}
import os

from trajectory import Client

client = Client()
tid = client.trajectories.create().tid
response = client.chat.completions.create(
    model="policy",
    messages=[{"role": "user", "content": os.environ["TASK_INSTRUCTIONS"]}],
    x_trajectory_id=tid,
)
```

In a managed evaluation or training run, the run selects the actor model; changing the
request's `model` does not select a different actor. Generation settings come from the
run's `options`, set when you start an [evaluation](https://docs.trajectory.ai/guides/evaluations.md#1-start-a-baseline)
or [training run](https://docs.trajectory.ai/guides/training.md#training-options): `max_output_tokens_per_turn` replaces
the request's `max_tokens`, and the run's `temperature`, `top_p`, `top_k`, and
`reasoning_effort` take precedence over request values. Your harness decides whether one
response is enough or whether the model needs more turns.

When using Trajectory for a source-fixed judge or other auxiliary model, use an
organization-key client and a separate trajectory. `trajectories.create()` with the
managed session key reuses the actor's trajectory. Pass the separate trajectory's ID
with each auxiliary model request: `x_trajectory_id` in the Trajectory SDK or
`X-Trajectory-Id` in direct HTTP calls. Keep the organization key out of actor-visible
tools.

You can give the model tools if your harness supports executing tool calls and returning
their results to the model. Each tool describes an action the model can request, such as
running a shell command or reading a file. Include any software those tools need in the
image and provide the harness code that executes them.

## Report the grade through the SDK

Your verifier decides how to grade the model's work. It might compare an answer with an
expected value, run a test suite or inspect files and running services.

For an existing spec, trace the score field or formula its evaluation reader uses.
Keep the native scale and grading rule, including partial credit when that is the
reported score; a successful grader command alone does not identify the score.
Once your verifier produces a valid numeric `grade`, report it against the same
trajectory ID, then mark the attempt complete:

```python harness.py theme={null}
client.trajectories.log_reward(
    tid,
    name="reward_accuracy",
    value=grade,
)
client.trajectories.complete(tid)
```

Rewards default to `[0, 1]`. For a different scale, declare its bounds when logging:

```python harness.py theme={null}
client.trajectories.log_reward(
    tid,
    name="score",
    value=-0.5,
    min_value=-1,
    max_value=1,
)
```

The effective reward is the weighted average of components scaled to `[0, 1]` by their
declared bounds. Use `weight=0` for a secondary score you want to inspect without
changing that reward. Retrieve components with `include_reward_components=True`;
`trajectory.reward` alone can be zero without a native grade. The [source-grader guide](https://docs.trajectory.ai/guides/source-graders.md) shows a worked mapping and
grader-output event.

These snippets show the SDK calls to place in your harness; they are not a complete
program. `complete()` defaults to `ENV_DONE`, which means the attempt finished normally.
When it did not, pass the reason. For an unrecovered model, environment, or verifier
failure, report `ERROR` even if an incidental grader output exists. Do not publish
that output as reward or substitute zero:

```python harness.py theme={null}
client.trajectories.complete(tid, termination_reason="ERROR")
```

[Termination reasons](https://docs.trajectory.ai/guides/termination-reasons.md) explains reason selection, including
cut-off replies and how endings affect training and evaluation.

A completed attempt can have an incorrect answer. An error means the harness could not
produce a valid grade. Upload any outputs and logs you need before completion, because
the sandbox may be cleaned up afterward.

## Package the harness in an image

Include your harness, task files and dependencies in the image. This Dockerfile installs
the latest SDK and copies your application files into `/app`:

```dockerfile Dockerfile theme={null}
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir --upgrade trajectory-sdk
COPY . /app/
```

Here, `run_command="python /app/harness.py"` assumes `harness.py` is in that build
context. Check that paths in your task instructions and commands match where the
Dockerfile copies files. For the `/workspace/...` paths above, you could put the task
files under `benchmark/workspace/` locally and add `COPY workspace/ /workspace/` to the
Dockerfile. `COPY . /app/` alone would put them under `/app/workspace/` instead.

A build context holds at most 4,096 files. For a larger task set, archive the task files,
`COPY` the archive, and unpack it in a `RUN` step. `ADD` accepts only a single `http://` or
`https://` URL, not local files or archives.

Add the tools and verifier dependencies your task needs, keeping held-out tests
out of any environment the model can inspect. Keep credentials out of the image.
Test that separation through the actor's actual file and shell tools; hiding a path
from its prompt does not make the path private.

`DockerfileBuild` uploads files under the Dockerfile's directory. Scope that context
to at most 4,096 files and 3 GiB; use `.dockerignore` or a smaller runtime directory
when the checkout is larger. `COPY` paths are relative to that context.

For harnesses that run Docker containers, such as Harbor, install and verify the Docker
daemon, CLI and Compose during the image build. On Debian Trixie:

```dockerfile Dockerfile theme={null}
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
       docker.io docker-cli docker-compose \
    && docker --version \
    && docker compose version \
    && dockerd --version \
    && rm -rf /var/lib/apt/lists/*
```

## Validate one task before uploading

For quick debugging, run `start_task_diagnostic()` on one representative task. A diagnostic
reports each task's status and, for a failed task, its failure: a code, plus the rollout and
trajectory IDs, termination reason, and a redacted summary of any harness exception when the
task got that far:

```bash theme={null}
pip install --upgrade trajectory-sdk
export TRAJECTORY_API_KEY="YOUR_TRAJECTORY_API_KEY"
```

```python theme={null}
import time
from pathlib import Path

from trajectory import Client, TaskSpec
from trajectory.lib import DockerfileBuild, start_task_diagnostic

client = Client()
task = TaskSpec(
    name="repair-config",
    runtime=DockerfileBuild("Dockerfile"),
    run_command="python /app/harness.py",
    env_vars={"TASK_INSTRUCTIONS": "Fix the invalid configuration in /workspace/config.json."},
)
diagnostic = start_task_diagnostic(
    client,
    task,
    agent_name="YOUR_AGENT_NAME",
    root=Path("benchmark"),
    options={"max_output_tokens_per_turn": 4096},
)
while True:
    status = client.diagnostics.get_status(diagnostic.benchmark_diagnostic_id)
    if status.status in {"completed", "failed", "cancelled"}:
        break
    time.sleep(5)
result = client.diagnostics.get_diagnostics(diagnostic.benchmark_diagnostic_id)
print(result.status, result.task_counts, result.failure)
for task_result in result.tasks:
    print(
        task_result.task_id,
        task_result.status,
        task_result.termination_reason,
        task_result.reward,
        task_result.failure,
    )
```

The helper handles uploads and runtime builds without an existing spec ID.
Its `timeout_seconds` bounds only the image-build wait, not task execution.
`ImageRef(...)` and `RuntimeRef(...)` also work if you already have an image or runtime.
Check each task's failure and the run-wide `result.failure` before moving on.

A diagnostic makes one attempt per task on the API's default diagnostic model, reported as
`result.base_model_slug`; pass `base_model_slug` to choose another. `options` takes the
settings an [evaluation](https://docs.trajectory.ai/guides/evaluations.md#1-start-a-baseline) accepts, except the sample
counts, which the diagnostic sets. Pass the model and settings you will evaluate and train
with, so the diagnostic generates replies the same way. Each task reports its
`trajectory_id`, the `termination_reason` it ended with, and its `reward`, which is null when
the trajectory ended before the harness recorded one.

## Upload the spec and build its image

Put your Dockerfile, harness and task files in a local `benchmark/` directory.
The example below uploads the `task` defined above to your chosen agent:

```python upload.py theme={null}
from pathlib import Path

from trajectory import BenchmarkSpec, Client
from trajectory.lib import DockerfileBuild, push

client = Client()
benchmark = BenchmarkSpec(
    name="Configuration repair",
    runtime=DockerfileBuild("Dockerfile"),
    tasks=[task],
)
uploaded = push(
    client, benchmark, agent_name="YOUR_AGENT_NAME", root=Path("benchmark"), build_images=True
)
print(uploaded.bench_id)
```

This uploads your tasks and files and waits for the image build. If registration or
building fails, `push()` raises `BenchmarkOperationError`, which you can import from
`trajectory.lib`; inspect its `failures` for the affected runtime and build error.

For an upload that may outlive your client, use `start_push()` and save its returned
operation `id`. A local timeout does not stop ingestion: reconnect with
`get_operation(client, operation_id)`, then inspect `refresh()` and `pages("failure")`
before retrying. A failed runtime blocks evaluation of that spec version;
correct it and upload a new version rather than creating a second, partial spec.

To use this reconnectable form instead of the `push()` call above:

```python theme={null}
from trajectory.lib import start_push

operation = start_push(
    client, benchmark, agent_name="YOUR_AGENT_NAME", root=Path("benchmark"), build_images=True
)
print(operation.id)
result = operation.result(timeout=45 * 60)
print(result.status.bench_id)
```

Uploading does not run the tasks. After the diagnostic passes, run a small
[evaluation](https://docs.trajectory.ai/guides/evaluations.md) and confirm a recorded grade. A graded zero is valid;
use held-out tasks when comparing checkpoints.

`push()`, `start_push()`, and `start_task_diagnostic()` accept `upload_timeout` in seconds
for blob store uploads, defaulting to `3600` (one hour). Pass `upload_timeout=None` to disable
blob store timeouts. These apply to individual HTTP operations against the blob store,
separately from API request timeouts, image-build waits, and signed-URL expiry. They do not
set a total upload deadline.

`push()` and `start_push()` also accept `upload_max_retries`, the retries for each upload request
(default `32`, within 4.5 minutes of waiting). See [Errors and retries](https://docs.trajectory.ai/reference/errors.md).

You can share a runtime across tasks or set `runtime` on individual tasks:

| Runtime definition | What you provide |
| - | - |
| `DockerfileBuild("Dockerfile")` | A Dockerfile and the files in its directory. |
| `ImageRef("registry.example.com/agent@sha256:...")` | An existing image pinned to a digest. |
| `RuntimeRef("YOUR_RUNTIME_ID")` | A runtime already registered in your organization. |

Import these runtime helpers from `trajectory.lib`. For secret task inputs, use
`SecretRef(secret_ref="SECRET_NAME")` in `env_vars` instead of packaging secret values.

## Upload and diagnose the full spec

Expand `BenchmarkSpec.tasks` to the full task set and upload it with `push()`. Once
`push()` finishes, call `client.diagnostics.start_benchmark()` using
that upload's `bench_id`. It runs up to 10 tasks, one attempt each, starting with one
task per runtime, and reports the same task outcomes and failures as the single-task
diagnostic. You need not repeat the single-task diagnostic for an unchanged runtime.

```python theme={null}
import time

diagnostic = client.diagnostics.start_benchmark(
    bench_id=uploaded.bench_id, options={"max_output_tokens_per_turn": 4096}
)
while True:
    status = client.diagnostics.get_status(diagnostic.benchmark_diagnostic_id)
    if status.status in {"completed", "failed", "cancelled"}:
        break
    time.sleep(5)

report = client.diagnostics.get_diagnostics(diagnostic.benchmark_diagnostic_id)
print(report.status, report.task_counts, report.failure)
for task_result in report.tasks:
    print(
        task_result.task_id,
        task_result.status,
        task_result.termination_reason,
        task_result.reward,
        task_result.failure,
    )
```

Inspect the run-wide failure and every task's outcome, ending, and reward. If the diagnostic fails, fix the reported problem, upload the revised
full spec, and diagnose that version. After it passes, run an
[evaluation](https://docs.trajectory.ai/guides/evaluations.md) to confirm a recorded reward before
[training](https://docs.trajectory.ai/guides/training.md).

Evaluation and training accept `agent_name` and `benchmark_name` together to select the
latest version by name. Use `bench_id` when you need a specific version; spec
diagnostics require the ID returned by the upload helper.

## Update your spec

To upload a revised spec, use the same spec name and agent and include the
full set of tasks you want in the new version. Earlier versions remain available.
Evaluation and training by name select the latest version, so finish a baseline/checkpoint
comparison before uploading another version under the same name.

## Tips for adapting existing specs

Your harness implements the deadlines, grading environments and evidence uploads below.

* **Bring the verifier as well as the task environment.** Check that its scripts,
  dependencies and expected paths are available when grading starts. A ready image
  does not prove the verifier can run; test one task before launching the full set.
* **Give the agent and verifier separate time limits.** When the agent runs out of
  time, stop its model calls and tools, then grade the work completed so far. Leave
  the verifier its full allowance and account for both phases in the rollout timeout.
* **Follow the spec's grading rules.** Preserve resource limits, startup commands
  and environment settings. Some specs grade in the same running container;
  others require applying the model's changes to a fresh one. If the model has shell
  or file access, your harness must keep held-out tests outside the environment those
  tools can inspect. Uploading an image does not automatically provide this separation.
* **Keep enough evidence to explain a result.** Save the generated output, verifier logs
  and error details as artifacts before completing the trajectory. Report missing logs
  separately from missing grades, and state any changes to the original spec.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.