What are we training?
For each ticket, the model assigns probabilities to["billing", "technical"].
The answer it selects is the one with the higher probability. A billing ticket’s
label is [1, 0]; a technical ticket’s is [0, 1]. Keep that label out of the
input you send to the model.
RLCD adjusts those probabilities using Brier loss:
0.5 * sum((p[i] - y[i]) ** 2), averaged over decisions. Here p is the model’s
probability vector and y is the label. The trainer recomputes probabilities at
the captured decision position and differentiates through all the choices.
Labels can be soft, too: [0.8, 0.2] is valid. Values must be finite, between zero
and one, and sum to one. A scalar reward can’t replace this vector.
The SDK supports text and image inputs. Training takes exactly one trainable
decision step per trajectory; groups of one are fine. Repeated rollouts and
variation in scalar reward aren’t needed.
Inference limits and workarounds
Use onetype="choice" question per request, with 2–26 distinct, nonempty choice
values. predicate and score aren’t supported question types.
For yes/no, supply those two choices and read the probability of yes. For a numeric
rating, supply the rating levels as choices and calculate their probability-weighted
average yourself. The API returns a choice and probabilities, not a numeric score.
Objects and arrays must be serialized to text, such as json.dumps(data), and sent
as input: there is no input_json part. More than 26 choices won’t work in one
request. The renderer’s labels are the single tokens A–Z.
1. Write the harness
Save the following asbenchmark/harness.py. It reads the ticket and label from
environment variables, makes a decision, and records both the training labels and
an accuracy reward.
harness.py
Client() during the run. Use model="policy"
to call the model being trained or evaluated. Here, “policy” is an alias for that
model. This calls POST /v1/decisions with X-Trajectory-Id.
The response’s result has choice, probabilities, and confidence. Confidence
is adjusted for the number of choices: (max_probability - 1/N) / (1 - 1/N).
It isn’t the selected answer’s probability.
For a pending request, use the recovery procedure in errors and retries.
Keep the same idempotency key; don’t start a second logical decision.
The labels to log
For our billing ticket, logdecision.outcome.0=1 and decision.outcome.1=0.
In general, log exactly one decision.outcome.<index> per choice, in the original
zero-based order, before completing the trajectory. Set each component’s weight
to zero and bounds to [0, 1].
Zero weight keeps labels out of the reported reward. Training reads their raw
values anyway. Accuracy is a separate reward for comparing evaluations and doesn’t
affect the Brier gradient. With only zero-weight components, even a correct answer
has an aggregate reward of zero.
Don’t substitute the chosen answer’s reward or a precomputed Brier score for the
labels. For retries, reuse each component’s distinct reward_id through
extra_body. Leaving it out creates a new ID on each call.
2. Upload a decision spec
Put this Dockerfile beside the harness:Dockerfile
upload.py next to the benchmark/ directory. Replace YOUR_AGENT_ID with
an agent you’ve created or chosen.
upload.py
benchmark_type="decision". A call to the decision API
won’t turn a generative spec into a decision spec. Only decision specs accept RLCD.
These two tickets exercise the workflow; they can’t tell you whether training
improves a model. Use your own dataset and an independent held-out split for that.
Before a full run, check readiness and diagnostics. If you add tools
to this harness, keep them from reading EXPECTED_CATEGORY.
3. Evaluate, then train
Use the uploaded spec ID and a model that supports decision training. This prints the spec/model’s available algorithms, bounds, and defaults, then starts a baseline:train_on_* filters enabled. Groups with identical scalar rewards
can still produce Brier updates. batch_size counts eligible task groups per update.
Check optimizer steps and checkpoints,
not just reward. Cancel if the run exceeds your budget.
Evaluate the checkpoint on the baseline’s
spec version and held-out split. Compare your accuracy reward together with the
completed and failed counts. The two-ticket example isn’t evidence of improvement.
To use a checkpoint from another algorithm, set algorithm="rlcd" and
load_weights_only=True in options. Full resume needs a compatible algorithm;
loading weights alone gives you fresh training state. See resume.
Image decisions
Use a model whoseinput_modalities includes image. Upload the image
to the same trajectory, then put its artifact ID in the ordered list of input parts: