Outcomes, calibration and uncertainty

What does a score mean?

Use task_outcome(result) to retrieve the declared primary outcome. Retain its key, raw value, normalised value, scoring window and calibration status. When a result has several targets, this function returns a mapping by target ID.

A matching calibration defines a signed adjustment:

adjusted = (raw - null_mean) / (upper_bound - null_mean)

Chance performance maps to zero; the task bound maps to one. Values below the null remain negative. Different task units and null policies remain different quantities, so this transformation does not authorise a cross-task competence average.

Freeze the relevant null

Binary probes declare analytic chance 0.5. Physical tasks use exactly 1,024 independent trajectories of their declared random-action reference. The supplied calibration plans can be resolved before any expensive execution:

uv run brainlesslab check python/plans/calibration-tracking.toml
uv run brainlesslab check python/plans/calibration-pong.toml
uv run brainlesslab check python/plans/calibration-cartpole-easy.toml

A successful check reports the calibration protocol; it does not produce a calibration. Execute the chosen plan through brainlesslab run and retain its complete record. Supply that record with the run command’s --calibration option only when its signature matches the target.

The signature includes task options, initialisation, total horizon, warm-up, action cadence, numerical/RNG policy, backend, architecture, runtime fingerprint and world-source hashes. It excludes reservoir kind, count, topology and input gain. Calibration and evaluated world streams must be disjoint. A random-action null answers a different question from blind input, shifted input or a plasticity ablation.

Report uncertainty separately

Summaries first average trial values within each independent model block. Adjusted intervals independently resample model blocks and calibration trajectories, then recompute the ratio. The default bootstrap uses 2,000 draws and linear 2.5% and 97.5% quantiles. Raw and null intervals remain separate reported quantities.

Fewer than two model blocks gives no model interval. Invalid denominators make the adjusted interval unavailable; the procedure does not discard those bootstrap draws. A missing physical calibration leaves the raw outcome available and the adjusted value unavailable.

Decode observations offline

decode fits a ridge decoder to sparse recorded emitted activity. It is a diagnostic analysis of recoverable task information, not a trained online controller. It supports the six single-episode capacity families and excludes physical tasks and action-dependent reversal sessions.

The default split is 256 fit trials, 128 validation trials and 256 evaluation trials per independent fixed-wiring block. Scaling uses fit trials only; validation selects the ridge penalty separately at each observation point. A permutation null shuffles fit and validation labels separately and leaves evaluation labels unchanged.

See the decoding procedure for recording, ownership and split checks. For task performance comparisons use operations; for runtime speed use performance.