Skip to content
BrainlessLab BrainlessLab BrainlessLab

Compare conditions

This tutorial compares Falandays with a random-output control on Tracking, Pong, and Wall. It is a software-scale example, not the canonical Falandays benchmark.

Inspect and validate the comparison

Open plans/examples/benchmark_core.toml. Each case names conditions on one task and declares its random control as the baseline.

Terminal window
julia --project=. bin/brainlesslab.jl check \
  plans/examples/benchmark_core.toml

Validation requires conditions in one case to share their task, block structure, horizon, warm-up, reset, construction scope, root seed, streams, and aggregation.

Run the benchmark

Terminal window
julia -t auto --project=. bin/brainlesslab.jl run \
  plans/examples/benchmark_core.toml --root records

Inspect two tables:

  • summary/statistics.csv gives raw and normalised intervals for each condition;
  • summary/contrasts.csv gives paired differences from the declared baseline.

Keep Tracking, Pong, and Wall separate. Their raw outcomes differ, and normalisation does not turn them into one competence scale.

Use an anchor-only case when you want to describe one condition without a comparator. Such a case has exactly one condition and omits baseline.

Source: plans/examples/benchmark_core.toml, src/operations/Benchmark.jl.