Adapt an existing grader

Read the source evaluator’s result reader before choosing a reward. Identify the score it publishes, its range, and whether partial credit contributes. A successful grader command does not establish which score the benchmark reports. Select the score in the task harness, after grading. Do not choose it in the upload script, clamp it, or turn a missing score into zero:

import math

def select_score(scores: dict[str, object], key: str) -> float:
    if key not in scores:
        raise ValueError(f"Missing score {key!r}; available: {sorted(scores)}")
    value = scores[key]
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise ValueError(f"Score {key!r} must be numeric")
    score = float(value)
    if not math.isfinite(score):
        raise ValueError(f"Score {key!r} must be finite")
    return score

report = grade_answer(model_answer, expected_answer)
score = select_score(report, "accuracy")  # Use the source reader's primary field.
client.trajectories.log_reward(tid, name="accuracy", value=score)

Parse the source grader’s documented format first if it returns strings, booleans, or nested data. Declare the actual bounds in log_reward when they differ from [0, 1]. Publish either the native total once or components that reproduce it, not both. A wrong answer can have a valid zero; an unrecovered model, environment, or grader error should end as ERROR without a published reward, even if an incidental verifier result exists. For a secondary score, weight=0 makes it visible without changing the effective reward. Use it only when the harness owns reward publication; even a zero-weight component can prevent automatic grading. An event can hold grader details without affecting reward:

fraction = select_score(report, "criteria_fraction")
client.trajectories.log_reward(
    tid, name="criteria_fraction", value=fraction, weight=0,
)
client.trajectories.log_event(
    tid, event_id="rubric", name="rubric",
    payload={"criteria_fraction": fraction},
)

Keep private grader references out of events. Retrieve the trajectory with include_reward_components=True to check names, values, bounds, and weights. Compare wrong, partial, and correct outputs against the source evaluator, including its aggregation. Then run a task diagnostic through the packaged runtime and inspect the reward, ending, and failures. Repeat for each distinct grading schema.

Get started

Build and improve

Production learning

Reference

Adapt an existing grader