Evaluation

EvaluationSpec is the outer repetition protocol for every operation. It is separate from neural frames and world ticks.

For a Tracking target, one valid protocol is:

evaluation = EvaluationSpec(
    blocks=20,
    trials_per_block=1,
    horizon=2200,
    warmup=200,
    construction_scope=:trial,
    reset=:full,
    root_seed=51001,
    aggregate=:mean,
)

This protocol scores 2,000 ticks after warm-up, which meets Tracking’s declared minimum.

Independent units

A block or trial is usually the independent randomised unit. Agents, nodes, samples, and ticks within one world do not multiply the sample size.

Use blocks to pair declared conditions under shared nuisance randomisation. Use trials_per_block when a block contains repeated trials by design.

Construction and reset

construction_scope controls whether topology is constructed once per evaluation, block, or trial. reset controls what is reset between trials. Generic operation plans currently support only reset=:full; check rejects :body_environment and :none before a run starts because their state-retention hooks are not implemented.

warmup runs before recording and scoring. The scored duration is horizon - warmup. Choose both values from the task timescale, not from compute convenience alone.

Random streams

The default root-seed policy derives topology and world streams. Declare another named stream only when an implementation consumes it. Records retain every declared realised seed.

Separate calibration, development, variance-pilot, confirmation, and robustness seed ledgers. A reproducible seed does not make reused development data independent evidence.

Aggregation

Aggregation combines declared trial values inside an operation. It does not authorise a cross-task aggregate. Keep task-native results and missing or failed trials visible.

Source: src/core/Specifications.jl, src/api/Composition.jl.