Skip to main content
Train a decision model to choose among fixed answers using labeled examples. Your harness calls the decision API and records an outcome for each choice; RLCD uses those outcomes to train the model’s probabilities with Brier loss. The examples below use support-ticket routing to show the workflow.

What are we training?

For each ticket, the model assigns probabilities to ["billing", "technical"]. The answer it selects is the one with the higher probability. A billing ticket’s label is [1, 0]; a technical ticket’s is [0, 1]. Keep that label out of the input you send to the model. RLCD adjusts those probabilities using Brier loss: 0.5 * sum((p[i] - y[i]) ** 2), averaged over decisions. Here p is the model’s probability vector and y is the label. The trainer recomputes probabilities at the captured decision position and differentiates through all the choices. Labels can be soft, too: [0.8, 0.2] is valid. Values must be finite, between zero and one, and sum to one. A scalar reward can’t replace this vector. The SDK supports text and image inputs. Training takes exactly one trainable decision step per trajectory; groups of one are fine. Repeated rollouts and variation in scalar reward aren’t needed.

Inference limits and workarounds

Use one type="choice" question per request, with 2–26 distinct, nonempty choice values. predicate and score aren’t supported question types. For yes/no, supply those two choices and read the probability of yes. For a numeric rating, supply the rating levels as choices and calculate their probability-weighted average yourself. The API returns a choice and probabilities, not a numeric score. Objects and arrays must be serialized to text, such as json.dumps(data), and sent as input: there is no input_json part. More than 26 choices won’t work in one request. The renderer’s labels are the single tokens A–Z.

1. Write the harness

Save the following as benchmark/harness.py. It reads the ticket and label from environment variables, makes a decision, and records both the training labels and an accuracy reward.
harness.py
Trajectory supplies credentials to Client() during the run. Use model="policy" to call the model being trained or evaluated. Here, “policy” is an alias for that model. This calls POST /v1/decisions with X-Trajectory-Id. The response’s result has choice, probabilities, and confidence. Confidence is adjusted for the number of choices: (max_probability - 1/N) / (1 - 1/N). It isn’t the selected answer’s probability. For a pending request, use the recovery procedure in errors and retries. Keep the same idempotency key; don’t start a second logical decision.

The labels to log

For our billing ticket, log decision.outcome.0=1 and decision.outcome.1=0. In general, log exactly one decision.outcome.<index> per choice, in the original zero-based order, before completing the trajectory. Set each component’s weight to zero and bounds to [0, 1]. Zero weight keeps labels out of the reported reward. Training reads their raw values anyway. Accuracy is a separate reward for comparing evaluations and doesn’t affect the Brier gradient. With only zero-weight components, even a correct answer has an aggregate reward of zero. Don’t substitute the chosen answer’s reward or a precomputed Brier score for the labels. For retries, reuse each component’s distinct reward_id through extra_body. Leaving it out creates a new ID on each call.

2. Upload a decision spec

Put this Dockerfile beside the harness:
Dockerfile
Save upload.py next to the benchmark/ directory. Replace YOUR_AGENT_ID with an agent you’ve created or chosen.
upload.py
The important field is benchmark_type="decision". A call to the decision API won’t turn a generative spec into a decision spec. Only decision specs accept RLCD. These two tickets exercise the workflow; they can’t tell you whether training improves a model. Use your own dataset and an independent held-out split for that. Before a full run, check readiness and diagnostics. If you add tools to this harness, keep them from reading EXPECTED_CATEGORY.

3. Evaluate, then train

Use the uploaded spec ID and a model that supports decision training. This prints the spec/model’s available algorithms, bounds, and defaults, then starts a baseline:
Wait for it to finish and check the captured decisions, outcomes, and accuracy. A successful container build doesn’t check label logging. Then try a two-step training run:
Leave the three train_on_* filters enabled. Groups with identical scalar rewards can still produce Brier updates. batch_size counts eligible task groups per update. Check optimizer steps and checkpoints, not just reward. Cancel if the run exceeds your budget. Evaluate the checkpoint on the baseline’s spec version and held-out split. Compare your accuracy reward together with the completed and failed counts. The two-ticket example isn’t evidence of improvement. To use a checkpoint from another algorithm, set algorithm="rlcd" and load_weights_only=True in options. Full resume needs a compatible algorithm; loading weights alone gives you fresh training state. See resume.

Image decisions

Use a model whose input_modalities includes image. Upload the image to the same trajectory, then put its artifact ID in the ordered list of input parts:
The labels and Brier loss work the same way. Training retains the captured image inputs.

Troubleshooting

See decision training troubleshooting for rejected outcomes, missing capture metadata, and training configuration issues.