Evaluation reference
request_specs.pybuilds the exact API request shapes and native worker jobs without sending or executing them.run_api.pyconnects to the recorded Gateway models or a provider/model endpoint you configure, and saves fresh responses.connections.pyreads provider connection settings without reading credentials or making requests.prepare_native.pyprepares the all-model download specification and validated GPU jobs offline.score_variants.pycompares fresh original and altered inputs using the exact manipulation sets.score_fresh.pyscores fresh API answers or GPU worker receipts against the selected questions.reference/worker_entry.pyis the entrypoint for the native GPU worker package. Run it in an isolated Linux CUDA container.reference/provenance.jsonrecords the copied source hashes and worker revision.- The pinned dependency files are under
workers/. workers/Dockerfilebuilds the core and large-model environments from those dependency files.
For ordinary verification, use python3 baseline.py verify; no evaluation environment is needed. See reproduction instructions for setup and fresh-run commands.
To use a provider directly or run another model, follow Connect a model. Its example connection files describe the endpoint, request format and model ID; API keys are supplied separately.
Both fresh-run clients accept --case, --all or --cases-file. Use data/binary_subset.csv to select the 214-question subset, or an available-input JSONL to select only questions whose text you have. All selected inputs are validated before execution.
The scorer compares fresh answers with the saved baseline over the same selected IDs. Add --output-dir .local/comparison to save its report and a CSV of changed answers or statuses. It does not update the reference results.