Initial weights and plasticity
Research · Overview of the initialisation study and the Falandays–SORN follow-up
Changing a network’s weights as it runs can improve its performance, but the benefit depends on where it starts and what it is doing. In our Falandays experiments, frozen weights worked better on Pong across the tested initialisation settings. On Tracking, the advantage changed direction with the starting weight scale.
The investigation began with an observation from Marc Bacvanski: smaller initial weights could allow frozen Falandays networks to outperform adapting ones. This raises a question we want to pursue: how much does plasticity help a network reach a useful operating range, and how much does it help once it gets there?
Two findings in Falandays networks
We compared networks whose connection weights could change with networks whose weights stayed fixed. Both retained the same rule for adapting each node’s activity target. We repeated this comparison across five initial weight scales.
On Pong, frozen weights performed better at every tested scale. The network controls a paddle to intercept a ball. Freezing its weights increased the mean hit rate by roughly 14–19 percentage points compared with allowing them to adapt. This pattern appeared in the original sweep and again in a larger follow-up with fresh seeds and standardised sensory and output connections.
On Tracking, the preferred policy depended on the initial weights. Here the network follows a moving stimulus around a circle. Frozen weights performed better at an intermediate starting scale; adapting weights performed better at the two largest scales tested. The two smallest scales gave no clear advantage to either policy.
Together, these results make initialisation part of the plasticity question. The same weight-update rule can help or hinder performance under different starting conditions and tasks.
A comparison with SORN
We also ran SORN networks on the same two tasks. Their clearest pattern involved changes to firing thresholds: on Pong, threshold adaptation reduced performance at small recurrent-strength settings and improved it at an intermediate setting. The effects of freezing weights were generally smaller, and the Tracking results gave little evidence of a consistent adaptation benefit.
This comparison broadens the question to the different processes grouped under “plasticity”. In SORN, we varied both initial and maintained recurrent strength, so this is a comparison of operating conditions rather than an exact repeat of the Falandays initialisation test.
The next question
These experiments compare networks that adapt or stay frozen from the beginning. They leave open whether an initial period of adaptation could establish useful dynamics that later updates disrupt.
The next direct test would let paired networks adapt for the same initial period, then freeze one while the other continues adapting. Separate tests for weights and activity targets or thresholds would show which ongoing updates help. Recording their activity alongside task performance would help explain why.
Read the further work note or contribute a related investigation.
Study design and full results
The follow-up compares two models, two tasks, five strength settings and four adaptation policies. Each condition uses 32 independent wiring blocks, with four fully reset trials per block. All 10,240 trials completed. The results are exploratory; the reported intervals are pointwise 95% block-bootstrap intervals.
The full comparison report contains all policy curves, numerical contrasts, uncertainty estimates, calibration details and reproducibility downloads. It also examines how weight adaptation interacts with activity-target or threshold adaptation.
The original four-task report retains the earlier sweep and its construction review. The follow-up used fresh seeds, more independent blocks and standardised connections; differences between studies combine those changes.