Compare across four tasks

What is being compared?

The benchmark helper constructs reservoir and reference-null conditions for Tracking, Pong, CartPole and delayed cue. Each pair shares its declared world opportunity. The output is a set of task coordinates, not a common competence score.

from brainlesslab import EvaluationSpec, NodeSpec, benchmark, resolve

plan = benchmark(
    NodeSpec("sorn"), count=32,
    evaluation=EvaluationSpec(blocks=2, trials_per_block=4, root_seed=130),
)
for target in resolve(plan).targets:
    print(target.target.id, target.target.controller,
          target.horizon, target.outcome_key)

Expect eight targets: a reservoir and random-action reference for each task. With no shared horizon override, every task retains its own default. This snippet checks configuration only.

benchmark concerns task performance. To measure warm runtime, compilation, recording or transfer costs, use performance.

Read coordinates in native units

Tracking reports mean cosine alignment, Pong reports hit fraction, CartPole reports world steps survived and delayed cue reports response accuracy. A physical reference-null target is a matched comparison; it is not automatically a frozen 1,024-trajectory calibration.

Use matching calibration when reporting adjusted values. Report raw values, scoring windows, null policy and intervals as separate fields. Within-task paired raw contrasts answer the condition difference directly. Do not average adjusted coordinates into an overall cognitive ranking.

Separate available software from study evidence

Falandays and SORN are active model families. The capacity-probe library adds experimental task opportunities. Neither model availability nor a successful development record satisfies a frozen historical confirmation protocol.

Historical benchmark tables and accepted contributions keep their original source revisions, plans and evidence states. Query the research catalogue before citing them.

Start with task contracts, choose a model and run a small reproducible profile before a full benchmark.