> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trajectory.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Train a model

> Train on a spec (benchmark) and pick a checkpoint.

Training uses a ready [spec](https://docs.trajectory.ai/guides/spec.md) and its rewards. Start from an [evaluation baseline](https://docs.trajectory.ai/guides/evaluations.md).

## 1. Choose a model

```python theme={null}
from trajectory import Client

client = Client()
bench_id = "YOUR_BENCHMARK_ID"
catalog = client.training.list_options(bench_id=bench_id)

for model in catalog.models:
    print(model.base_model_slug, model.options)
```

Use a returned `base_model_slug`. Each entry describes that model's supported options,
bounds, and defaults. A computed default has `default_kind="computed"` and a `null`
`default`; leave the option unset to let the server calculate it.

## 2. Start

```python theme={null}
run = client.training.create(
    bench_id=bench_id,
    base_model_slug="YOUR_SUPPORTED_TRAINING_MODEL_SLUG",
    options={"num_steps": 20},
)
print(run.training_run_id)
```

Training and evaluation take `RunOptions` fields through `options`; a standalone
evaluation accepts [a subset](https://docs.trajectory.ai/guides/evaluations.md#1-start-a-baseline).
Omit `options`, or pass `{}`, to use the selected model's defaults. Only set options
advertised by that mode's `list_options` response; training and evaluation can have
different bounds.

`batch_size` counts eligible task groups per optimizer step, and `group_size` counts
trajectories within a group. For a short test run, lower both; default concurrency scales
with them. Nothing here caps total trajectories, so plan a budget before you start. See
[training options](#training-options) for every setting and
[reduce trajectory spend](#reduce-trajectory-spend) for the ones that control token use.

## 3. Watch progress and pick a checkpoint

```python theme={null}
import time

while True:
    training = client.training.runs.retrieve(run.training_run_id)
    progress = client.training.runs.progress(run.training_run_id)
    print(training.status, progress.completed_steps, progress.total_steps)
    if training.status in {"succeeded", "failed", "cancelled"}:
        break
    time.sleep(30)

if training.status != "succeeded":
    raise RuntimeError(f"Training {training.status}: {training.failure}")
```

An ID means the run was accepted. A pending run with zero completed steps is waiting;
increasing step counts show progress. Watch attempted and graded trajectories as well.
If trajectories accumulate without an update, inspect their endings and reward components;
cancel a test run that has exceeded its planned budget. Check completed steps
and checkpoints before concluding that training updated the model. See
[termination reasons](https://docs.trajectory.ai/guides/termination-reasons.md) for which endings training admits.

For a small test run, inspect trajectories and group counts:

```python theme={null}
trajectories = list(client.trajectories.list(training_run_id=run.training_run_id))
print(len(trajectories), [(item.trajectory_id, item.status) for item in trajectories])
for step in client.training.runs.list_step_summaries(run.training_run_id).items:
    print(step.step, step.no_signal_groups, step.used_groups, step.pending_groups)
```

`no_signal_groups` counts completed groups with identical rewards, which give zero
advantage in group-relative training. When these groups are filtered out, training
needs additional trajectories to fill the batch.

After the run succeeds, inspect its rewards and choose a checkpoint.
`list_held_out_rewards` returns the validation reward for each step:

```python theme={null}
for point in client.training.rewards.list_history(run.training_run_id):
    print(point.step, point.avg_reward)
for point in client.training.rewards.list_held_out_rewards(run.training_run_id):
    print(point.step, point.held_out_reward, point.round_completed,
          point.completed_samples, point.expected_samples)

checkpoints = client.training.checkpoints.list(run.training_run_id)
checkpoint = max(checkpoints, key=lambda c: c.step_index)
print(checkpoint.step_index, checkpoint.checkpoint_id, checkpoint.base_model_slug)
```

Check `round_completed`, and `completed_samples` against `expected_samples`, before
comparing validation means. The agent's **Training** tab shows run progress and checkpoints.

[Evaluate that checkpoint](https://docs.trajectory.ai/guides/evaluations.md#3-score-a-checkpoint) before you deploy it.

## 4. Resume from a checkpoint

```python theme={null}
resumed = client.training.create(
    bench_id=bench_id,
    base_model_slug=checkpoint.base_model_slug,
    parent_checkpoint_id=checkpoint.checkpoint_id,
    options={"num_steps": 20},
    display_name="Resumed training",
)
print(resumed.training_run_id)
```

The checkpoint must match the selected base model. Use `parent_checkpoint_id` for
both resumed training and checkpoint evaluation. Resuming restores the optimizer, LR
schedule, and global step; set `load_weights_only` to `true` to start from the checkpoint's
weights with fresh training state.

## Training options

Pass these keys in `options`. The defaults below apply to every supported model unless
noted; `list_options` reports the bounds and defaults for the model you select. A default
written as `null` → a value is computed when you leave the option unset. With the default
`batch_size` of 32 and `group_size` of 8, that resolves to 512 concurrent training
trajectories and a minimum of 4 eligible trajectories per group.

### Optimization

| Option | Default | What it controls |
| - | - | - |
| `num_steps` | `100` | Number of optimizer steps to run (1–100,000). |
| `algorithm` | `gspo` | Training algorithm; omit to preserve the model default. One of `gspo` or `grpo`. |
| `batch_size` | `32` | Eligible task-instance groups per optimizer update (1–256). |
| `group_size` | `8` | Trajectories generated for each training instance; GSPO and GRPO require at least 2 (at most 32). |
| `min_trajectories_per_group` | `null` → `group_size // 2` | Minimum eligible trajectories per training group. `null` uses half of `group_size`, rounded down (1–32). |
| `learning_rate` | `1e-4` | Base optimizer learning rate (1e-8–1e-2). |
| `lr_warmup_steps` | `15` | Reported training steps used for warmup, including skipped optimizer updates. |
| `lr_hold_steps` | `135` | Training steps to hold the base rate after warmup, including skipped updates, before cosine decay. |
| `lr_min_ratio` | `0.1` | Cosine decay floor as a fraction of the base learning rate (0–1). |
| `max_staleness` | `3` | Maximum optimizer steps a training trajectory's starting policy may trail the current policy (0–100). Older in-flight trajectories are cancelled and older completed trajectories are discarded; `0` trains only on trajectories started from the current policy. |
| `load_weights_only` | `false` | Load parent checkpoint weights with fresh training state. Defaults to `false`, which resumes the optimizer, LR schedule, and global step. With validation enabled, evaluate the starting policy, run validation when `global_step % val_interval_steps == 0`, and evaluate the final policy on normal completion. Resuming preserves the global step and validation cadence. |
| `density_aware_training` | `false` | Train each task at four output-token budgets, with `group_size` trajectories per budget. Turns on `train_on_truncation`, which must stay `true` except when resuming a checkpoint with its training state. `batch_size` sets the total number of task groups per batch; each budget counts as a separate group. [Learn more](https://www.trajectory.ai/field-notes/intelligence-density). |

The learning rate rises linearly to `learning_rate` over `lr_warmup_steps`, holds until step
`lr_warmup_steps + lr_hold_steps`, then follows a cosine curve down to
`learning_rate × lr_min_ratio` at the final step.

![Learning rate by training step: a linear rise to the base rate by step 15, flat until step 150, then a cosine curve down to 10% of the base rate at the final step of a 400-step example. A default 100-step run ends during the flat section.](https://mintcdn.com/trajectory/U_ViD_ffmPwHpDb1/images/lr-schedule-light.svg?fit=max\&auto=format\&n=U_ViD_ffmPwHpDb1\&q=85\&s=6d9127a7c8f533d6f81c85f37c4fe1f2)

With the defaults, decay starts at step 150, so a default 100-step run ends at the base rate.
To use the decay, set `num_steps` above 150 or lower `lr_hold_steps`.

### Training filters

| Option | Default | What it controls |
| - | - | - |
| `train_on_truncation` | `false`; `true` with `density_aware_training` | Include trajectories that hit token or turn limits in training, with reward 0. Defaults to `false`, or `true` with `density_aware_training`; set to `true` to discourage reward hacking through truncated submissions. |
| `train_on_timeout` | `false` | Include timed-out trajectories in training, with reward 0. Defaults to `false`; set to `true` to discourage reward hacking through timed-out submissions. |
| `train_on_error` | `false` | Train on errored trajectories, including execution errors, grading errors, and missing grades, with reward 0. |
| `train_on_all_fail` | `false` | Train on groups whose eligible rewards are all at or below zero; `false` excludes them. |
| `train_on_all_pass` | `false` | Train on groups whose eligible rewards are all at or above one; `false` excludes them. |
| `train_on_zero_advantage` | `false` | Train on groups with identical eligible rewards; `false` excludes them. |

Reward 0 applies to training only; the recorded trajectory reward stays as logged. See
[termination reasons](https://docs.trajectory.ai/guides/termination-reasons.md).

### Trajectories

| Option | Default | What it controls |
| - | - | - |
| `max_concurrent_trajectories` | `null` → `batch_size × group_size × 2` | Maximum active training trajectories (1–1,024). `null` uses `batch_size × group_size × 2`, allowing 2× batch oversampling for throughput and asynchronous RL. Recommended: set above `batch_size × group_size` to keep collection ahead of training. |
| `max_turns_per_trajectory` | `200` | Maximum agent and environment turns allowed in each trajectory (1–500). |
| `max_output_tokens_per_turn` | `null` → model maximum | Maximum output tokens per model response. `null` uses the model's maximum, bounded by remaining context. |
| `max_output_tokens_per_trajectory` | `null` → model maximum context length | Maximum output tokens across one trajectory, including after context compaction (1–2,097,152). `null` uses the model's maximum supported context length. |
| `trajectory_timeout_seconds` | `7200` | Maximum execution seconds per trajectory, including grading (1–36,000). Excludes queueing and sandbox startup. Collection timeouts must cover this budget. |
| `batch_timeout_seconds` | `9600` | Maximum wall-clock time to assemble one training batch (1–38,400). The clock starts when collection begins and does not reset as groups or in-flight trajectories arrive. Must cover trajectory execution and is not adjusted automatically. |

### Sampling

| Option | Default | What it controls |
| - | - | - |
| `reasoning_effort` | `null` → lowest supported thinking level | Model reasoning effort. `null` selects the lowest supported thinking level; `off` disables thinking where supported. Levels: `minimal`, `low`, `medium`, `high`, `xhigh`, `max`, depending on the model. Qwen3.5-4B, Qwen3.6-35B-A3B, Qwen3.5-397B-A17B, and Nemotron-3.5-Lightning-30B-A3B support only on/off; `null` selects `on`. `list_options` reports each model's `choices`; other values return 422. |
| `temperature` | `1.0` | Sampling temperature; defaults to 1.0 (0–2). |
| `top_p` | `1.0` | Nucleus sampling probability; `null` preserves the model default (0–1). |
| `top_k` | `-1` | Sample from the top k tokens; `-1` disables the limit, and `null` preserves the model default. |

### Validation during training

| Option | Default | What it controls |
| - | - | - |
| `val_enabled` | `true` | Enable validation during training. |
| `val_interval_steps` | `1` | Training steps between validations; `0` disables validation (0–100). When using `0`, omit `val_enabled`; an explicit `true` is rejected. |
| `val_num_tasks` | `50` | Maximum distinct tasks per evaluation. Standalone evals run every task in the selected scope when omitted. |
| `val_trajectories_per_task` | `1` | Number of trajectories generated for each task in a validation round. |
| `val_max_concurrent_trajectories` | `50` | Maximum number of validation trajectories running concurrently. |
| `val_final_wait_timeout_seconds` | `7200` | Maximum seconds to wait for outstanding validation rounds at training end; `0` stops waiting immediately. |
| `val_reasoning_effort` | `null` → training setting | Validation reasoning effort. `null` inherits training defaults. Supports the same model-specific levels and on/off options as `reasoning_effort`. |
| `val_temperature` | `null` → training setting | Validation sampling temperature (0–2). `null` inherits training defaults. |
| `val_top_p` | `null` → training setting | Validation nucleus sampling probability (0–1). `null` inherits training defaults. |
| `val_top_k` | `null` → training setting | Validation top-k token limit. `null` inherits training defaults; `-1` disables it. |
| `val_max_output_tokens_per_turn` | `null` → training setting | Maximum output tokens per turn in each validation trajectory. `null` inherits training defaults. |
| `val_max_output_tokens_per_trajectory` | `null` → training setting | Maximum output tokens across one validation trajectory. `null` inherits training defaults. |
| `val_max_turns_per_trajectory` | `null` → training setting | Maximum agent and environment turns per validation trajectory. `null` inherits training defaults. |
| `val_trajectory_timeout_seconds` | `null` → training setting | Maximum execution seconds per validation trajectory, including grading. `null` inherits training defaults. |

## Reduce trajectory spend

Training is asynchronous: trajectories keep running up to `max_concurrent_trajectories`,
including while an optimizer step is in progress. Each step trains on `batch_size` complete
groups that pass filtering. Groups whose trajectories all pass, all fail, or tie carry no
learning signal, so by default they are discarded after using tokens.

* **Concurrency.** The default oversamples by 2×. Lower `max_concurrent_trajectories` to
  spend less on trajectories the trainer may not use, but keep it above
  `batch_size × group_size` so collection stays ahead of training.
* **Staleness.** Trajectories that started more than `max_staleness` steps before the
  current step are discarded: running ones are cancelled and finished ones are skipped.
  Lower values keep training closer to on-policy but discard more work. `0` cancels every
  trajectory still running when a step finishes.
* **Validation.** With the defaults, a 100-step run validates 101 times on up to 50
  tasks, about 5,000 extra trajectories. Set `val_interval_steps` to `5` or `10`,
  lower `val_num_tasks`, or set `val_interval_steps` to `0` and
  [evaluate checkpoints](https://docs.trajectory.ai/guides/evaluations.md#3-score-a-checkpoint) after training.

```python theme={null}
run = client.training.create(
    bench_id=bench_id,
    base_model_slug="YOUR_SUPPORTED_TRAINING_MODEL_SLUG",
    options={
        "num_steps": 50,
        "max_concurrent_trajectories": 384,
        "val_interval_steps": 10,
        "val_num_tasks": 25,
    },
)
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.