Rewards are flat at zero
A zero can mean a graded wrong answer, or no grade at all.trajectory.reward can
be zero without a native grade, and a step summary’s incorrect_count counts finished rollouts
with reward 0, so neither shows that a grader ran. A harness that catches a grader failure and
then calls complete() records ENV_DONE with no grade.
Retrieve an affected trajectory with include_reward_components=True. Use
client.trajectories.list(training_run_id=...) or eval_run_id=... to find attempts.
Inspect termination_reason, reward, and reward_components together.
Check that the verifier and its dependencies are in the image. When grading fails,
complete with
ERROR and publish no reward. Options such as train_on_all_fail and
train_on_zero_advantage change which groups training admits; they do not repair grading.
Upload the fix as a new version and diagnose one task.
Check that it reports the ending and reward you expect, and that its reward components are present.
Training attempts keep growing with no completed steps
Attempts run up tomax_concurrent_trajectories even when no group can be trained
on. Groups with identical rewards carry no learning signal under GSPO and GRPO. When
these groups are filtered out, training keeps collecting attempts to fill the batch.
Compare client.training.runs.progress(run_id) with
client.training.runs.list_step_summaries(run_id). Inspect completed_steps,
trajectory_count, no_signal_groups, used_groups, and termination_reasons.
used_groups counts consumed groups with differing rewards, not every optimizer input.
Check completed steps, trainer rewards, and checkpoints before concluding that the model
updated. If every group has no signal, start with
flat rewards.
Stopping your client does not stop a run. Cancel a validation run that has exceeded
its planned budget with client.training.runs.cancel(run_id), then fix the cause.
See Reduce trajectory spend for the settings that
bound attempts before you restart.
Poll until the run is terminal; a cancellation request alone is not confirmation.
On the next run, check optimizer progress before scaling up.
TimeoutError: Timed out waiting for ...; server processing continues
The client stopped waiting for a benchmark upload. Ingestion continues on the server,
so running the upload again can create a second version.
Reconnect with the operation ID in the message, then read its state before retrying:
status = operation.result().status. If it succeeded, use its bench_id. If
it failed, continue with BenchmarkOperationError.
Confirm status.status is succeeded and status.ready is True if you requested
image builds.
To keep the ID for next time, upload with start_push() and save operation.id before waiting.
BenchmarkOperationError or a failed image build
Registration or a runtime build failed. The exception’s failures name each affected
resource and its error. A failed runtime blocks evaluation of that benchmark version.
Correct the problem, then upload a new version with the same benchmark name and agent.
Do not create a second, partial benchmark.
The new upload returns a new bench_id. Diagnose one task before running it at scale.
Runtime context ... exceeds the 4,096-file limit
DockerfileBuild uploads every file under the Dockerfile’s directory, up to 4,096
files and 3 GiB.
Keep only that runtime’s files in its directory, or exclude the rest with
.dockerignore at the benchmark root passed to push() or start_push(), with patterns
relative to that root. For a larger task set, archive the files, COPY the archive, and unpack it in
a RUN step.
After uploading, diagnose one task to check that its files are available.
A model request returns an empty reply
After a platform turn, token, or context limit, model requests may return an empty reply withmetadata.trajectory_termination_reason: "trajectory_cancelled".
Stop calling the model and retrieve the trajectory to read its recorded reason. If the
harness must grade a reply that fills its per-step allowance, leave room between
max_output_tokens_per_turn and max_output_tokens_per_trajectory.
Check client.trajectories.retrieve(tid).termination_reason for the platform’s reason.
A harness keeps retrying a rejected attempt
A harness with its own retries can lose the service’s decision when it replaces an SDK error with its own. Preserve the SDK error’s status, headers, and body. In particular, pass throughTrajectory-Should-Retry: false so the harness does not retry that attempt.
Verify that the harness does not retry a response with that header.
HARNESS_FAILED before the first model call
Read the task diagnostic’s failure and logs. A trajectory can have no steps if imports,
task setup, or a model-call argument failed before inference. Retrieve the trajectory’s
error and inspect its rollout diagnostic; do not assume an empty transcript is a model
failure. Save diagnostic events and artifacts before the harness exits; see
diagnostic artifacts. After fixing the cause, rerun that task
with the same model and options, and check its ending and native grade.
Git reports detected dubious ownership
Image-built repositories can be owned by a different user from the runtime harness.
Set the expected ownership during image construction, or trust the specific repository
with git config --global --add safe.directory /workspace/repo in the runtime user context.
Use the actual repository path rather than trusting every directory. Confirm the harness
and grader can run their Git commands as that user.