Skip to content
BrainlessLab BrainlessLab BrainlessLab

Experiments

Experiment code and scientific evidence are different things. A registered protocol may be runnable before it has enough independent replication, a complete result packet, or a defensible interpretation. This index makes that status explicit.

Terminal window
julia --project=. experiments/run.jl --list
julia --project=. experiments/run.jl NAME [key=value ...]

Named experiments are separate from the general RunConfig API. Each named study defines its own options and output schema; inspect its source before overriding defaults.

Evidence levels

Tracking parameter sweep is the only study on this site currently backed by committed per-seed result JSON, manifests, generated figures, and matching experiment code. It includes a one-at-a-time parameter sweep and a leak × lrate_wmat factorial over 100 reused seed identities per cell.

The page reports descriptive grid results and their limits. In particular, the largest observed cell is a discovery estimate, not a held-out optimum, and the checked-in manifests do not capture a dirty patch or full package lockfile.

Exploratory protocols

  • Freeze-plasticity onset is a runnable registered protocol, but no committed result packet supports a precise onset claim.
  • Social foraging records the question, controls, and currently defensible estimands. The earlier small, ad-hoc sweeps are not treated as confirmatory evidence.
  • Criticality controls records a narrow measurement problem and protocol constraints. Existing plots and spot checks are exploratory.
  • search_efficiency and forage_intrinsic_drive_controls are registered working studies. Their local or incomplete outputs are not promoted here as findings.

Standalone finite-size scripts under experiments/ remain development protocols unless they are registered, documented, completed with adequate replication, and promoted with a coherent result packet. Files sharing an output directory do not automatically constitute one reproducible study.

Required result packet

A result presented as evidence should provide:

  1. Question and estimand — one predeclared primary quantity and its direction.
  2. Design — conditions, controls, independent unit, paired-seed scheme, horizon, exclusions, censoring, and stopping rule.
  3. Raw data — per-seed rows sufficient to recompute every reported aggregate.
  4. Uncertainty — intervals or tests at the independent-unit level, with multiplicity handled when selecting across a grid.
  5. Provenance — resolved config, full Git SHA, dirty state or patch, Julia version, dependency/Manifest identity, and analysis schema version.
  6. Completion state — unambiguous RUNNING, FAILED, or DONE marker; partial cells must not appear complete.
  7. Interpretation and limitations — claims restricted to what the controls identify.

Use immutable run directories for materially different protocols. Checkpoint keys must include every simulation-affecting setting; otherwise a resumed run can silently mix old cells with a new manifest.

Writing style

Public experiment pages follow a short structure: question, design, primary result, interpretation, methods/provenance, and limitations. Proposed work and roadmap material do not belong in a results page. An exploratory page may document a method without announcing a finding.

Source: experiments/ for protocols and experiments/results/ for committed result packets.