Benchmark submission operations

This workflow requires the asynchronous ingestion backend, worker job, queue, and recovery schedule to be deployed and enabled. Do not release this SDK workflow before that rollout.

from trajectory.lib import benchmarks
from trajectory.lib.benchmark_operations import BenchmarkPartialFailureError

operation = benchmarks.submit(client, manifest, root=path, idempotency_key="submission-1")
status = operation.refresh()
result = operation.result(timeout=1800)

# Reconnect in another process.
operation = benchmarks.get_operation(client, operation.id)

# An append is one submission, with its own operation and idempotency key.
appended = benchmarks.submit(client, manifest, root=path, bench_id=bench_id)

# Image builds remain explicit. Registration-only success does not mean ready to run.
ready = benchmarks.submit(client, manifest, root=path, build_images=True).result()

submit() transfers artifacts and task parts before returning an accepted operation. It does not wait for registration or image builds. The optional progress callback receives uploaded_files, uploaded_bytes, total_files, and total_bytes during transfer. Polling timeouts stop local waiting; they do not cancel server processing. Reuse an idempotency key only for the same metadata, task order, and artifact contents. Different input is rejected. Task parts are at most 4 MiB and 256 tasks. An individual task exceeding the byte limit must be reduced or move its large data into an artifact. Files are at most 256 MiB. Each task’s runtime/mount inputs are limited to 4,096 files and 256 MiB total.

try:
    result = operation.result()
except BenchmarkPartialFailureError as error:
    result = error.partial_result
    for failure in error.failures:
        print(failure.resource_id, failure.code, failure.message, failure.retryable)
        if failure.resource_type == "runtime":
            for task in result.affected_tasks(
                failure.resource_id, cursor=failure.affected_tasks_cursor
            ):
                print(task["name"])

for task in result.tasks():
    print(task["name"], task["task_id"])

Tasks, runtimes, failures, and affected tasks are paginated iterators. A partial failure preserves successful task registration and successful runtimes. ready is true only for a successful submission that explicitly requested builds and completed all required tasks and runtimes. push() and append() remain blocking conveniences with their existing successful return types. They now use submission operations and raise a specific partial-failure exception instead of silently returning an incomplete benchmark. Use submit() to control waiting and reconnect across processes. Default train/test assignment retains the 85/15 ratio but uses deterministic hash ordering; explicit task splits are unchanged. The lower-level upload_benchmark() and append_benchmark_tasks() helpers retain their small-request behavior. Large manifests or artifact maps automatically use operation transport, so the legacy full-manifest registration and diff requests remain bounded. Large holodeck submissions are explicitly rejected by this SDK path. The asynchronous append path uploads its artifacts and skips unchanged task writes on the server; it does not perform the legacy unchanged-artifact diff optimization.

Getting Started

SDK

HTTP API

Platform

Benchmark submission operations