> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trajectory.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Find a symptom or error message, its cause, and how to confirm the fix.

Search this page for the error text or the symptom you see. For how the SDK reports request
failures, see [Errors and retries](https://docs.trajectory.ai/reference/errors.md).

## Rewards are flat at zero

A zero can mean a graded wrong answer, or no grade at all. `trajectory.reward` can
be zero without a native grade, and a step summary's `incorrect_count` counts finished rollouts
with reward 0, so neither shows that a grader ran. A harness that catches a grader failure and
then calls `complete()` records `ENV_DONE` with no grade.

Retrieve an affected trajectory with `include_reward_components=True`. Use
`client.trajectories.list(training_run_id=...)` or `eval_run_id=...` to find attempts.
Inspect `termination_reason`, `reward`, and `reward_components` together.

| What you see | What it means | What to do |
| - | - | - |
| No components | No grade was recorded. | Fix the grader or harness. |
| Components with a real score of zero | The answers were graded wrong. | Read the transcript and check the task and grader. |
| `ERROR` or another non-`ENV_DONE` ending | The attempt did not finish normally. | See [Termination reasons](https://docs.trajectory.ai/guides/termination-reasons.md). |

Check that the verifier and its dependencies are in the image. When grading fails,
complete with `ERROR` and publish no reward. Options such as `train_on_all_fail` and
`train_on_zero_advantage` change which groups training admits; they do not repair grading.

Upload the fix as a new version and [diagnose one task](https://docs.trajectory.ai/guides/spec.md#validate-one-task-before-uploading).
Check that it reports the ending and reward you expect, and that its reward components are present.

## Training attempts keep growing with no completed steps

Attempts run up to `max_concurrent_trajectories` even when no group can be trained
on. Groups with identical rewards carry no learning signal under GSPO and GRPO. When
these groups are filtered out, training keeps collecting attempts to fill the batch.

Compare `client.training.runs.progress(run_id)` with
`client.training.runs.list_step_summaries(run_id)`. Inspect `completed_steps`,
`trajectory_count`, `no_signal_groups`, `used_groups`, and `termination_reasons`.

`used_groups` counts consumed groups with differing rewards, not every optimizer input.
Check completed steps, trainer rewards, and checkpoints before concluding that the model
updated. If every group has no signal, start with
[flat rewards](#rewards-are-flat-at-zero).

Stopping your client does not stop a run. Cancel a validation run that has exceeded
its planned budget with `client.training.runs.cancel(run_id)`, then fix the cause.

See [Reduce trajectory spend](https://docs.trajectory.ai/guides/training.md#reduce-trajectory-spend) for the settings that
bound attempts before you restart.

Poll until the run is terminal; a cancellation request alone is not confirmation.
On the next run, check optimizer progress before scaling up.

## `TimeoutError: Timed out waiting for ...; server processing continues`

The client stopped waiting for a benchmark upload. Ingestion continues on the server,
so running the upload again can create a second version.

Reconnect with the operation ID in the message, then read its state before retrying:

```python theme={null}
from trajectory import Client
from trajectory.lib import get_operation

client = Client()
operation = get_operation(client, "YOUR_OPERATION_ID")
status = operation.refresh()
print(status.status, status.stage, status.bench_id)
for failure in operation.pages("failure"):
    print(failure["resource_id"], failure["code"], failure["message"])
```

If it is still running, wait with `status = operation.result().status`. If it succeeded, use its `bench_id`. If
it failed, continue with [`BenchmarkOperationError`](#benchmarkoperationerror-or-a-failed-image-build).

Confirm `status.status` is `succeeded` and `status.ready` is `True` if you requested
image builds.

To keep the ID for next time, upload with `start_push()` and save `operation.id` before waiting.

## `BenchmarkOperationError` or a failed image build

Registration or a runtime build failed. The exception's `failures` name each affected
resource and its error. A failed runtime blocks evaluation of that benchmark version.

Correct the problem, then upload a new version with the same benchmark name and agent.
Do not create a second, partial benchmark.

The new upload returns a new `bench_id`. Diagnose one task before running it at scale.

## `Runtime context ... exceeds the 4,096-file limit`

`DockerfileBuild` uploads every file under the Dockerfile's directory, up to 4,096
files and 3 GiB.

Keep only that runtime's files in its directory, or exclude the rest with
`.dockerignore` at the benchmark root passed to `push()` or `start_push()`, with patterns
relative to that root. For a larger task set, archive the files, `COPY` the archive, and unpack it in
a `RUN` step.

After uploading, diagnose one task to check that its files are available.

## A model request returns an empty reply

After a platform turn, token, or context limit, model requests may return an empty
reply with `metadata.trajectory_termination_reason: "trajectory_cancelled"`.

Stop calling the model and retrieve the trajectory to read its recorded reason. If the
harness must grade a reply that fills its per-step allowance, leave room between
`max_output_tokens_per_turn` and `max_output_tokens_per_trajectory`.

Check `client.trajectories.retrieve(tid).termination_reason` for the platform's reason.

## A harness keeps retrying a rejected attempt

A harness with its own retries can lose the service's decision when it replaces an SDK
error with its own.

Preserve the SDK error's status, headers, and body. In particular, pass through
`Trajectory-Should-Retry: false` so the harness does not retry that attempt.

Verify that the harness does not retry a response with that header.

## `HARNESS_FAILED` before the first model call

Read the task diagnostic's failure and logs. A trajectory can have no steps if imports,
task setup, or a model-call argument failed before inference. Retrieve the trajectory's
`error` and inspect its rollout diagnostic; do not assume an empty transcript is a model
failure. Save diagnostic events and artifacts before the harness exits; see
[diagnostic artifacts](https://docs.trajectory.ai/guides/diagnostic-artifacts.md). After fixing the cause, rerun that task
with the same model and options, and check its ending and native grade.

## Git reports `detected dubious ownership`

Image-built repositories can be owned by a different user from the runtime harness.
Set the expected ownership during image construction, or trust the specific repository
with `git config --global --add safe.directory /workspace/repo` in the runtime user context.
Use the actual repository path rather than trusting every directory. Confirm the harness
and grader can run their Git commands as that user.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.