Skip to content
BrainlessLab BrainlessLab BrainlessLab

Experiments and evidence

Use ExperimentSpec when named conditions and one or more operations answer one scientific question. It records a stable ID, version, question, evidence state, limitations, metadata, conditions, and ordinary operation plans.

Evidence states

planned → exploratory → tuned → frozen → confirmed → promoted

Use retired for a withdrawn or superseded protocol. A complete record does not change the evidence state automatically.

Research sequence

  1. Verify implementation conformance, ports, reset, replay, and random streams.
  2. Calibrate task opportunity with an appropriate floor and ceiling.
  3. Explore behaviour and failure modes.
  4. Tune or train on development seeds.
  5. Estimate variance on fresh pilot blocks.
  6. Freeze the question, endpoint, controls, exclusions, and analysis.
  7. Run sealed confirmation once on untouched blocks.
  8. Test declared robustness conditions.
  9. Promote the complete protocol and records after scientific review.

Changing an endpoint, exclusion, condition, or analysis after viewing confirmation returns the study to development.

Controls and claims

Choose the control that answers the claim. Random action, blind input, matched sham, registered ablation, model baseline, replay, and oracle policies answer different questions.

Report what the code implements, what tests verify, what the experiment observes, and what the evidence supports as separate statements.

The Experiments catalogue lists versioned protocols. The Benchmarks catalogue lists task-specific benchmark records.

Accept a public run

Review and merge the protocol and software before anyone submits a public run. The run-only contribution must come from a clean source SHA that is reachable from main.

The contributor submission and maintainer replay remain two roles within one contribution. A replay is a portability and execution check, not a second independent experiment.

Automated checks validate records and compare roles. A human maintainer accepts or rejects the contribution after reviewing the protocol, provenance, comparison, and interpretation boundary. Acceptance does not change the experiment evidence state automatically.

Read Research records for the complete contribution path and publication limits.

Source: src/operations/Plans.jl, src/records/ExperimentIO.jl.