> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trajectory.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Train decision models

> Train decision models on labeled choices with RLCD and Brier loss.

Train a decision model to choose among fixed answers using labeled examples. Your
harness calls the decision API and records an outcome for each choice; RLCD uses
those outcomes to train the model's probabilities with Brier loss.

The examples below use support-ticket routing to show the workflow.

## What are we training?

For each ticket, the model assigns probabilities to `["billing", "technical"]`.
The answer it selects is the one with the higher probability. A billing ticket's
label is `[1, 0]`; a technical ticket's is `[0, 1]`. Keep that label out of the
input you send to the model.

RLCD adjusts those probabilities using Brier loss:
`0.5 * sum((p[i] - y[i]) ** 2)`, averaged over decisions. Here `p` is the model's
probability vector and `y` is the label. The trainer recomputes probabilities at
the captured decision position and differentiates through all the choices.

Labels can be soft, too: `[0.8, 0.2]` is valid. Values must be finite, between zero
and one, and sum to one. A scalar reward can't replace this vector.

The SDK supports text and image inputs. Training takes exactly one trainable
decision step per trajectory; groups of one are fine. Repeated rollouts and
variation in scalar reward aren't needed.

## Inference limits and workarounds

Use one `type="choice"` question per request, with 2–26 distinct, nonempty choice
values. `predicate` and `score` aren't supported question types.

For yes/no, supply those two choices and read the probability of yes. For a numeric
rating, supply the rating levels as choices and calculate their probability-weighted
average yourself. The API returns a choice and probabilities, not a numeric score.

Objects and arrays must be serialized to text, such as `json.dumps(data)`, and sent
as `input`: there is no `input_json` part. More than 26 choices won't work in one
request. The renderer's labels are the single tokens A–Z.

## 1. Write the harness

Save the following as `benchmark/harness.py`. It reads the ticket and label from
environment variables, makes a decision, and records both the training labels and
an accuracy reward.

```python harness.py theme={null}
import os

from trajectory import Client

client = Client()
tid = client.trajectories.create().tid
choices = ["billing", "technical"]
expected = os.environ["EXPECTED_CATEGORY"]
if expected not in choices:
    raise ValueError("EXPECTED_CATEGORY must be one of the choices")

response = client.decisions.create(
    model="policy",
    input=os.environ["TICKET_TEXT"],
    questions=[{
        "type": "choice",
        "name": "category",
        "instructions": "Route this support ticket to the appropriate team.",
        "choices": [{"value": value} for value in choices],
    }],
    x_trajectory_id=tid,
    idempotency_key=f"{tid}:category",
)
if response.result is None:
    raise RuntimeError("Decision did not return a result")

for index, value in enumerate(choices):
    client.trajectories.log_reward(
        tid,
        name=f"decision.outcome.{index}",
        value=float(value == expected),
        min_value=0,
        max_value=1,
        weight=0,
        extra_body={"reward_id": f"{tid}:outcome:{index}"},
    )

client.trajectories.log_reward(
    tid,
    name="accuracy",
    value=float(response.result.choice == expected),
    weight=1,
    extra_body={"reward_id": f"{tid}:accuracy"},
)
client.trajectories.complete(tid, termination_reason="ENV_DONE")
```

Trajectory supplies credentials to `Client()` during the run. Use `model="policy"`
to call the model being trained or evaluated. Here, "policy" is an alias for that
model. This calls `POST /v1/decisions` with `X-Trajectory-Id`.

The response's `result` has `choice`, `probabilities`, and `confidence`. Confidence
is adjusted for the number of choices: `(max_probability - 1/N) / (1 - 1/N)`.
It isn't the selected answer's probability.

For a pending request, use the recovery procedure in [errors and retries](https://docs.trajectory.ai/reference/errors.md).
Keep the same idempotency key; don't start a second logical decision.

### The labels to log

For our billing ticket, log `decision.outcome.0=1` and `decision.outcome.1=0`.
In general, log exactly one `decision.outcome.<index>` per choice, in the original
zero-based order, before completing the trajectory. Set each component's weight
to zero and bounds to `[0, 1]`.

Zero weight keeps labels out of the reported reward. Training reads their raw
values anyway. Accuracy is a separate reward for comparing evaluations and doesn't
affect the Brier gradient. With only zero-weight components, even a correct answer
has an aggregate reward of zero.

Don't substitute the chosen answer's reward or a precomputed Brier score for the
labels. For retries, reuse each component's distinct `reward_id` through
`extra_body`. Leaving it out creates a new ID on each call.

## 2. Upload a decision spec

Put this Dockerfile beside the harness:

```dockerfile Dockerfile theme={null}
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir trajectory-sdk
COPY harness.py /app/harness.py
```

Save `upload.py` next to the `benchmark/` directory. Replace `YOUR_AGENT_ID` with
an agent you've [created or chosen](https://docs.trajectory.ai/agent.md).

```python upload.py theme={null}
from pathlib import Path

from trajectory import BenchmarkSpec, Client, TaskSpec
from trajectory.lib import DockerfileBuild, push

client = Client()
benchmark = BenchmarkSpec(
    name="Support routing decisions",
    benchmark_type="decision",
    runtime=DockerfileBuild("Dockerfile"),
    tasks=[
        TaskSpec(
            name="billing-training",
            split="train",
            run_command="python /app/harness.py",
            env_vars={
                "TICKET_TEXT": "My card was charged twice for the same invoice.",
                "EXPECTED_CATEGORY": "billing",
            },
        ),
        TaskSpec(
            name="technical-held-out",
            split="test",
            run_command="python /app/harness.py",
            env_vars={
                "TICKET_TEXT": "The application crashes when I open settings.",
                "EXPECTED_CATEGORY": "technical",
            },
        ),
    ],
)
uploaded = push(
    client, benchmark, agent_id="YOUR_AGENT_ID", root=Path("benchmark"), build_images=True
)
print(uploaded.bench_id)
```

The important field is `benchmark_type="decision"`. A call to the decision API
won't turn a generative spec into a decision spec. Only decision specs accept RLCD.

These two tickets exercise the workflow; they can't tell you whether training
improves a model. Use your own dataset and an independent held-out split for that.
Before a full run, check [readiness and diagnostics](https://docs.trajectory.ai/guides/spec.md). If you add tools
to this harness, keep them from reading `EXPECTED_CATEGORY`.

## 3. Evaluate, then train

Use the uploaded spec ID and a model that supports decision training. This prints the
spec/model's available algorithms, bounds, and defaults, then starts a baseline:

```python theme={null}
from trajectory import Client

client = Client()
bench_id = "YOUR_DECISION_SPEC_ID"
model = "YOUR_SUPPORTED_TRAINING_MODEL_SLUG"
print(client.training.list_options(bench_id=bench_id, base_model_slug=model))

baseline = client.evals.create(bench_id=bench_id, base_model_slug=model)
print("Baseline:", baseline.eval_run_id)
```

[Wait for it to finish](https://docs.trajectory.ai/guides/evaluations.md) and check the captured decisions,
outcomes, and accuracy. A successful container build doesn't check label logging.
Then try a two-step training run:

```python theme={null}
run = client.training.create(
    bench_id=bench_id,
    base_model_slug=model,
    options={
        "algorithm": "rlcd",
        "num_steps": 2,
        "batch_size": 1,
        "group_size": 1,
        "min_trajectories_per_group": 1,
        "train_on_all_fail": True,
        "train_on_all_pass": True,
        "train_on_zero_advantage": True,
    },
)
print("Training:", run.training_run_id)
```

Leave the three `train_on_*` filters enabled. Groups with identical scalar rewards
can still produce Brier updates. `batch_size` counts eligible task groups per update.
Check [optimizer steps and checkpoints](https://docs.trajectory.ai/guides/training.md#3-watch-progress-and-pick-a-checkpoint),
not just reward. Cancel if the run exceeds your budget.

[Evaluate the checkpoint](https://docs.trajectory.ai/guides/evaluations.md#3-score-a-checkpoint) on the baseline's
spec version and held-out split. Compare your accuracy reward together with the
completed and failed counts. The two-ticket example isn't evidence of improvement.

To use a checkpoint from another algorithm, set `algorithm="rlcd"` and
`load_weights_only=True` in `options`. Full resume needs a compatible algorithm;
loading weights alone gives you fresh training state. See [resume](https://docs.trajectory.ai/guides/training.md#4-resume-from-a-checkpoint).

## Image decisions

Use a model whose `input_modalities` includes `image`. [Upload the image](https://docs.trajectory.ai/guides/images.md)
to the same trajectory, then put its artifact ID in the ordered list of input parts:

```json theme={null}
{
  "model": "policy",
  "input": [
    {"type": "text", "text": "Which category describes this picture?"},
    {"type": "image", "artifact_id": "YOUR_IMAGE_ARTIFACT_ID"}
  ],
  "questions": [{
    "type": "choice",
    "name": "category",
    "instructions": "Choose the matching category.",
    "choices": [{"value": "cat"}, {"value": "dog"}]
  }]
}
```

The labels and Brier loss work the same way. Training retains the captured image inputs.

## Troubleshooting

See [decision training troubleshooting](https://docs.trajectory.ai/reference/troubleshooting.md#decision-training-rejects-outcomes-or-captures)
for rejected outcomes, missing capture metadata, and training configuration issues.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.