Polarize
GitHub Get in touch

September 24, 2026 · Method · v9

Adaptive preregistration: predict first, then run the code

A protocol for simulations and research software in which the predictions, the pass criteria and the environment are fixed in git before anything a report will cite is allowed to run.

  • [A]
A page from Charles Darwin's 1837 notebook: the words
Charles Darwin, Notebook B (1837), page 36. Public domain. Desaturated for this site.

In short

Write the prediction and the pass criteria first, tag them, then run. Plans may change, but every change is dated, says whether the result was already known, and cannot turn a fail into a pass.

A simulation can be tuned until it agrees with you. Each choice is small: a seed, a threshold, an analysis window, a parameter nobody measured. None of them looks like cheating, and together they can manufacture any result.

Software makes this cheap. A new variant costs seconds, and the run that finally agrees looks exactly like one that was planned. The fix is the same one experimental science uses: write down what would count as success before looking, and keep a record that shows the order things happened in.

This is the protocol used for model experiments here. It adapts published work on adaptive preregistration for model-based research and on reporting deviations from a preregistered plan. It is written for simulations, analysis pipelines and research software, whoever or whatever is doing the running. The full protocol, with its sources and how deeply each was read, is in the protocol document.

The rules

  • A claim, or an exploratory label. Every piece of work, whether an app, a simulation, a tool or a calculator, starts with a falsifiable claim, settled in the author’s own words together with what would count against it. Work the author marks as exploratory may be built without one: it is recorded and labelled as exploratory, nothing it shows counts as a finding, and it gets its claim before its first experiment is preregistered.
  • No run before the preregistration. A run whose result a report will cite may not execute until PREREG.md is complete and tagged.
  • The environment is part of the plan. The preregistration names the model version it runs against, and a lockfile pins every dependency. Adding a dependency afterwards is a deviation.
  • Failure closes the version; it does not edit it. A failed experiment is closed with its failure written up, and the next attempt gets a new experiment ID. Every ID stays in a registry, so a pass after earlier failures is reported as one attempt among several.
  • Every departure from the plan is logged, dated, with whether the outcome was known at the time.
  • Exploratory work is quarantined. Anything not in the plan lives in exploratory/ and is labelled exploratory everywhere it is mentioned.

What makes it adaptive

Modelling is iterative, and a protocol that forbids iteration gets ignored. The rule is about order, not rigidity.

  • A plan may change between stages. Each stage says in advance which result leads to which change. The change is committed and tagged before the run it governs, and logged with the outcome marked as not yet known.
  • A plan may also change after a result is seen. It is still logged, with the outcome marked as known, and it can never be used to convert a fail into a pass.
  • An experiment that fails stays in the record. Nothing is deleted or cleaned up.

What the preregistration contains

Eleven sections, all written before any scoring run:

  • the question, as one mechanistic sentence;
  • risky, numeric, directional predictions, each with the result that would count against it;
  • pass and fail thresholds for each estimated quantity, with its uncertainty: a pass means the interval clears the threshold, not just the point estimate;
  • the outcomes that would close the line of work;
  • the model specification, with every parameter tagged as taken from literature, derived, or arbitrary;
  • the design: conditions, a fixed list of seeds, controls, exclusion rules, a justified number of runs with a stopping rule, and a sensitivity analysis for every arbitrary parameter. A pass that flips to a fail anywhere in a preregistered range is reported as not robust;
  • the environment, down to the execution backend and thread count, because the same seed does not give the same result everywhere;
  • what the authors already knew: which target data they had seen, and which targets the model was calibrated to rather than tested against;
  • any adaptive stages, every decision the specification left open, and how the plan was timestamped.

The results file answers the plan line by line. It also has to list the patterns the model failed to reproduce, what only worked because it was tuned, and the result of every control.

The record

Git holds the ordering. Each milestone is an annotated tag:

Tag Marks
model-vX.Y.Z a change to the model that alters its output
<EID>-prereg the complete plan, lockfile and configuration
<EID>-interim-N a plan revision, before the run it governs
<EID>-run the outputs and the results file
<EID>-closed the end of the experiment, pass or fail

Each experiment gets its own branch and draft pull request. Tagging a milestone posts a receipt to the pull request with the commit and a checksum of the plan and the lockfile. The comment’s timestamp comes from the hosting service, not from the machine that made the commit, so it shows the plan existed by then.

Rules for whoever runs it

The same rules apply to a person at the keyboard, a script, or a coding assistant.

  • Before running anything citable, confirm the plan is complete and tagged. If it is not, write it and stop for someone else to review.
  • Never edit the plan after it is tagged. Changes go in the deviations table.
  • Asked to “make it pass” or “try a few values and keep the best”, do the runs under exploratory/ and say plainly that they are not preregistered evidence.
  • State every open decision before the run that depends on it.
  • In every report, state how deeply each source was read: full text, abstract, background knowledge, or supplied by the user.
  • If anyone asks to skip a step, write the skip into the deviations table first, with the outcome-known field filled in honestly.
Historical drawing: the layers of the retina, with its receptor, bipolar and ganglion cells drawn one by one.
Santiago Ramón y Cajal, drawing of the retina (Tafel III). CC BY 4.0, Wellcome Collection. Inverted and toned for this site.

How it reaches every project

The protocol is maintained in one public repository, KIT Adaptive Preregistration, and installed into each project with a small command-line tool, kit_ap. It copies the protocol documents and tools into the project’s .agents/ folder and records the exact version in a lockfile. It also writes the rules into AGENTS.md, so coding assistants working in the repository read the same rules a person would.

Some of the rules are backed by code rather than left to good intentions:

  • an optional commit guard refuses outputs for an experiment that has no preregistration tag, and refuses any change to a plan after it is tagged (like any git hook, it can be bypassed);
  • where the tools support hooks, the current experiment’s plan status is surfaced on every session;
  • at the start of a session, the project checks whether the protocol has changed upstream, and reports it without updating on its own.

Limits

This protects the record, not the idea. A well-preregistered experiment on a wrong model is still wrong.

A local git history is not tamper-evident: tags can be moved unless they are signed and protected on the host. The pull-request receipt is a third-party clock but not an archive, since comments can be edited, and it can’t show that nothing ran before the plan. For outside credibility, the preregistration tag should also be deposited with an archive, and the plan should say which.

Nothing stops a new attempt after a failure. The registry only makes the attempts visible, so a reader can weigh a pass against them.

Most of the protocol is rules that people and tools are asked to follow. Only the commit guard and the hooks enforce anything, and the hooks only in tools that run them. The rest relies on the record making a violation visible afterwards.

Update — 2026-09-24

The kit’s repository is now public and linked from the section on how the protocol reaches each project.

After a methods review, the protocol was tightened, and this post was revised to match. It now describes: the registry of attempts, uncertainty in the pass criteria, a justified run count, a sensitivity rule fixed in advance, a section on prior knowledge and calibration (eleven sections, previously ten), and decision rules for adaptive stages. The limits section now says more about what the record can and can’t prove. Previously the post said a pass that depends on an arbitrary parameter does not count; the rule is now that a pass which flips within a preregistered range is reported as not robust.

Update — 2026-09-25

Recording coding-assistant conversations is no longer part of the protocol, and the kit no longer installs it. The post previously said each prompt and reply was posted to the experiment’s pull request and committed beside it; that paragraph and the matching line under “How it reaches every project” were removed. The kit’s repository was also renamed, and its links here point to the new address.

Update — 2026-09-30

The protocol gained a rule that comes before the others: every piece of work starts with a falsifiable claim, stated with what would count against it, before anything is built. Settling it means saying what the claim’s terms mean, how it will be measured, the smallest effect that would matter, and what a test of it assumes. The section above was previously titled “The five rules”. Preregistration now also works per piece of work: a project holding several apps, simulations, tools or calculators gives each its own claim, experiments and versions.

Update — 2026-10-07

The first rule changed. It previously said every piece of work starts with a falsifiable claim before anything is built. Work can now be marked exploratory instead: it is built from a short note of what is being built and what is already known, it is labelled exploratory, and nothing it shows counts as a finding. Its claim is settled before its first experiment is preregistered. Settling a claim before building now means stating the claim and what would count against it. What its terms mean, how it will be measured, the smallest effect that would matter and what a test of it assumes are stated in the first preregistration, not before building.

Version history 9 revisions

Generated from this site's git history. with Claude means the commit carries a co-author line for an Anthropic model: an LLM took part in that change, not necessarily in every word of it, and git cannot say which. The same history on GitHub →

  1. v9 2026-10-07 10:02 Post: update adaptive-preregistration post for kit_ap 179dc7b (0.9.0) Joshua Anderton with Claude Opus 5.5 +157 −4 words
    ## The rules
    - **ClaimA firstclaim, or an exploratory label.** Every piece of work, whether an app, a simulation, a tool or a calculator, starts with a falsifiable claim, settled in the author's own words together with what would count against it. Work the author marks as exploratory may be built without one: it is recorded and labelled as exploratory, nothing it shows counts as a finding, and it gets its claim before anythingits first experiment is builtpreregistered.
    - **No run before the preregistration.** A run whose result a report will cite may not execute until `PREREG.md` is complete and tagged.
    The protocol gained a rule that comes before the others: every piece of work starts with a falsifiable claim, stated with what would count against it, before anything is built. Settling it means saying what the claim's terms mean, how it will be measured, the smallest effect that would matter, and what a test of it assumes. The section above was previously titled "The five rules". Preregistration now also works per piece of work: a project holding several apps, simulations, tools or calculators gives each its own claim, experiments and versions.
    ## Update — 2026-10-07
    The first rule changed. It previously said every piece of work starts with a falsifiable claim before anything is built. Work can now be marked exploratory instead: it is built from a short note of what is being built and what is already known, it is labelled exploratory, and nothing it shows counts as a finding. Its claim is settled before its first experiment is preregistered. Settling a claim before building now means stating the claim and what would count against it. What its terms mean, how it will be measured, the smallest effect that would matter and what a test of it assumes are stated in the first preregistration, not before building.

    Commit 6960402 on GitHub →

  2. v8 2026-09-30 12:04 Post: claim first, and preregistration per piece of work Joshua Anderton with Claude Opus 5.5 +135 −1 words
    This is the protocol used for model experiments here. It adapts published work on adaptive preregistration for model-based research and on reporting deviations from a preregistered plan. It is written for simulations, analysis pipelines and research software, whoever or whatever is doing the running. The full protocol, with its sources and how deeply each was read, is in the [protocol document](https://github.com/polarizetech/adaptive-preregistration/blob/main/modules/prereg/protocols/PREREG_PROTOCOL.md).
    ## The five rules
    - **Claim first.** Every piece of work, whether an app, a simulation, a tool or a calculator, starts with a falsifiable claim, settled in the author's own words together with what would count against it, before anything is built.
    - **No run before the preregistration.** A run whose result a report will cite may not execute until `PREREG.md` is complete and tagged.
    Recording coding-assistant conversations is no longer part of the protocol, and the kit no longer installs it. The post previously said each prompt and reply was posted to the experiment's pull request and committed beside it; that paragraph and the matching line under "How it reaches every project" were removed. The kit's repository was also renamed, and its links here point to the new address.
    ## Update — 2026-09-30
    The protocol gained a rule that comes before the others: every piece of work starts with a falsifiable claim, stated with what would count against it, before anything is built. Settling it means saying what the claim's terms mean, how it will be measured, the smallest effect that would matter, and what a test of it assumes. The section above was previously titled "The five rules". Preregistration now also works per piece of work: a project holding several apps, simulations, tools or calculators gives each its own claim, experiments and versions.

    Commit ee47fcd on GitHub →

  3. v7 2026-09-25 13:23 Post: conversation recording is no longer part of the protocol Joshua Anderton with Claude Opus 5.5 +72 −39 words
    Each experiment gets its own branch and draft pull request. Tagging a milestone posts a receipt to the pull request with the commit and a checksum of the plan and the lockfile. The comment's timestamp comes from the hosting service, not from the machine that made the commit, so it shows the plan existed by then.
    When the work is done with a coding assistant, the conversation is recorded too: each prompt and reply is posted to the same pull request and committed beside the experiment, with secrets redacted.
    ## Rules for whoever runs it
    - an optional commit guard refuses outputs for an experiment that has no preregistration tag, and refuses any change to a plan after it is tagged (like any git hook, it can be bypassed);
    - where the tools support hooks, the current experiment's plan status is surfaced on every session, and assistant conversations are logged automatically;
    - at the start of a session, the project checks whether the protocol has changed upstream, and reports it without updating on its own.
    After a methods review, the protocol was tightened, and this post was revised to match. It now describes: the registry of attempts, uncertainty in the pass criteria, a justified run count, a sensitivity rule fixed in advance, a section on prior knowledge and calibration (eleven sections, previously ten), and decision rules for adaptive stages. The limits section now says more about what the record can and can't prove. Previously the post said a pass that depends on an arbitrary parameter does not count; the rule is now that a pass which flips within a preregistered range is reported as not robust.
    ## Update — 2026-09-25
    Recording coding-assistant conversations is no longer part of the protocol, and the kit no longer installs it. The post previously said each prompt and reply was posted to the experiment's pull request and committed beside it; that paragraph and the matching line under "How it reaches every project" were removed. The kit's repository was also renamed, and its links here point to the new address.

    Commit 9d79288 on GitHub →

  4. v6 2026-09-25 08:52 Posts: lead images for the spiking-network and adaptive-preregistration posts Joshua Anderton with Claude Opus 5.5 +60 −0 words
    into a pass.
    image: /assets/posts/darwin-notebook-b-tree.jpg
    image_alt: >-
    A page from Charles Darwin's 1837 notebook: the words "I think" above a hand-drawn
    branching tree of species, with handwritten notes beneath.
    image_credit: >-
    Charles Darwin, Notebook B (1837), page 36.
    <a href="https://commons.wikimedia.org/wiki/File:Darwin_Tree_1837.png" rel="noopener">Public domain</a>.
    Desaturated for this site.
    ---

    Commit e294818 on GitHub →

  5. v5 2026-09-25 07:44 Post: point the adaptive-preregistration links at the renamed repo Joshua Anderton with Claude Opus 5.5 +0 −2 words
    Software makes this cheap. A new variant costs seconds, and the run that finally agrees looks exactly like one that was planned. The fix is the same one experimental science uses: write down what would count as success before looking, and keep a record that shows the order things happened in.
    This is the protocol used for model experiments here. It adapts published work on adaptive preregistration for model-based research and on reporting deviations from a preregistered plan. It is written for simulations, analysis pipelines and research software, whoever or whatever is doing the running. The full protocol, with its sources and how deeply each was read, is in the [protocol document](https://github.com/polarizetech/kit-adaptive-preregistration/blob/main/modules/prereg/protocols/PREREG_PROTOCOL.md).
    ## The five rules
    ## How it reaches every project
    The protocol is maintained in one public repository, [KIT Adaptive Preregistration](https://github.com/polarizetech/kit-adaptive-preregistration), and installed into each project with a small command-line tool, `kit_ap`. It copies the protocol documents and tools into the project's `.agents/` folder and records the exact version in a lockfile. It also writes the rules into `AGENTS.md`, so coding assistants working in the repository read the same rules a person would.
    Some of the rules are backed by code rather than left to good intentions:

    Commit 8c23335 on GitHub →

  6. v4 2026-09-24 16:37 Post: update adaptive-preregistration post for kit v0.1.0 Joshua Anderton with Claude Opus 5.5 +321 −33 words
    Software makes this cheap. A new variant costs seconds, and the run that finally agrees looks exactly like one that was planned. The fix is the same one experimental science uses: write down what would count as success before looking, and keep a record that shows the order things happened in.
    This is the protocol used for model experiments here. It adapts published work on adaptive preregistration for model-based research and on reporting deviations from a preregistered plan. It is written for simulations, analysis pipelines and research software, whoever or whatever is doing the running. The full protocol, with its sources and how deeply each was read, is in the [protocol document](https://github.com/polarizetech/kit-adaptive-preregistration/blob/main/modules/prereg/protocols/PREREG_PROTOCOL.md).
    ## The five rules
    - **The environment is part of the plan.** The preregistration names the model version it runs against, and a lockfile pins every dependency. Adding a dependency afterwards is a deviation.
    - **Failure closes the version; it does not edit it.** A failed experiment is closed with its failure written up, and the next attempt gets a new experiment ID. Every ID stays in a registry, so a pass after earlier failures is reported as one attempt among several.
    - **Every departure from the plan is logged,** dated, with whether the outcome was known at the time.
    Modelling is iterative, and a protocol that forbids iteration gets ignored. The rule is about order, not rigidity.
    - A plan may change between stages. Each stage says in advance which result leads to which change. The change is committed and tagged *before* the run it governs, and logged with the outcome marked as not yet known.
    - A plan may also change after a result is seen. It is still logged, with the outcome marked as known, and it can never be used to convert a fail into a pass.
    ## What the preregistration contains
    TenEleven sections, all written before any scoring run:
    - the question, as one mechanistic sentence;
    - risky, numeric, directional predictions, each with the result that would count against it;
    - pass and fail thresholds for each estimated quantity, with anyits analyticuncertainty: expectationa derivedpass heremeans ratherthe thaninterval afterwardsclears the threshold, not just the point estimate;
    - the outcomes that would close the line of work;
    - the model specification, with every parameter tagged as taken from literature, derived, or arbitrary. A pass that depends on an arbitrary parameter does not count;
    - the design: conditions, a fixed list of seeds, controls and exclusion rules;
    - the design: conditions, a fixed list of seeds, controls, exclusion rules, a justified number of runs with a stopping rule, and a sensitivity analysis for every arbitrary parameter. A pass that flips to a fail anywhere in a preregistered range is reported as not robust;
    - the environment, down to the execution backend and thread count, because the same seed does not give the same result everywhere;
    - what the authors already knew: which target data they had seen, and which targets the model was calibrated to rather than tested against;
    - any adaptive stages, every decision the specification left open, and how the plan was timestamped.
    | `<EID>-closed` | the end of the experiment, pass or fail |
    Each experiment gets its own branch and draft pull request. Tagging a milestone posts a receipt to the pull request with the commit and a checksum of the plan and the lockfile. The comment's timestamp comes from the hosting service, not from the machine that made the commit, so it shows the plan existed by then.
    When the work is done with a coding assistant, the conversation is recorded too: each prompt and reply is posted to the same pull request and committed beside the experiment, with secrets redacted.
    Some of the rules are backed by code rather than left to good intentions:
    - an optional commit guard refuses outputs for an experiment that has no preregistration tag, and refuses any change to a plan after it is tagged (like any git hook, it can be bypassed);
    - where the tools support hooks, the current experiment's plan status is surfaced on every session, and assistant conversations are logged automatically;
    This protects the record, not the idea. A well-preregistered experiment on a wrong model is still wrong.
    A local git datehistory is not tamper-evident: tags can be moved unless they are signed and protected on the host. The pull-request receipt is a third-party clock but not an archive, since comments can be edited, and it can't show that nothing ran before the plan. For outside credibility, the preregistration tag should also be deposited with an archive, and the plan should say which.
    Nothing stops a new attempt after a failure. The registry only makes the attempts visible, so a reader can weigh a pass against them.
    Most of the protocol is rules that people and tools are asked to follow. Only the commit guard and the hooks enforce anything, and the hooks only in tools that run them. The rest relies on the record making a violation visible afterwards.
    The kit's repository is now public and linked from the section on how the protocol reaches each project.
    After a methods review, the protocol was tightened, and this post was revised to match. It now describes: the registry of attempts, uncertainty in the pass criteria, a justified run count, a sensitivity rule fixed in advance, a section on prior knowledge and calibration (eleven sections, previously ten), and decision rules for adaptive stages. The limits section now says more about what the record can and can't prove. Previously the post said a pass that depends on an arbitrary parameter does not count; the rule is now that a pass which flips within a preregistered range is reported as not robust.

    Commit f43b670 on GitHub →

  7. v3 2026-09-24 16:13 Post: link the now-public KIT Adaptive Preregistration repo Joshua Anderton with Claude Opus 5.5 +34 −0 words
    ## How it reaches every project
    The protocol is maintained in one public repository, [KIT Adaptive Preregistration](https://github.com/polarizetech/kit-adaptive-preregistration), and installed into each project with a small command-line tool, `kit_ap`. It copies the protocol documents and tools into the project's `.agents/` folder and records the exact version in a lockfile. It also writes the rules into `AGENTS.md`, so coding assistants working in the repository read the same rules a person would.
    Some of the rules are backed by code rather than left to good intentions:
    Most of the protocol is rules that people and tools are asked to follow. Only the commit guard and the hooks enforce anything, and the hooks only in tools that run them. The rest relies on the record making a violation visible afterwards.
    ## Update — 2026-09-24
    The kit's repository is now public and linked from the section on how the protocol reaches each project.

    Commit 19a8ba1 on GitHub →

  8. v2 2026-09-24 15:16 Retitle adaptive preregistration post and reframe away from AI Joshua Anderton with Claude Opus 5.5 +127 −163 words
    ---
    title: "Adaptive preregistration: whenpredict anfirst, agentthen runsrun the modelcode"
    description: >-
    A protocol for simulations and research software in which the predictions, the pass
    criteria and the environment are fixed in git before anything a language-modelreport agentwill cite is allowed to
    run anything a report will cite.
    allowed to run.
    date: 2026-09-24 12:00:00 -0600
    A simulation can be tuned until it agrees with you. Each choice is small: a seed, a threshold, an analysis window, a parameter nobody measured. None of them looks like cheating, and together they can manufacture any result.
    A coding agentSoftware makes this fastercheap. ItA cannew tryvariant twentycosts variants in the time it takes to read oneseconds, and itthe hasrun nothat stakefinally inagrees whichlooks exactly like one getsthat reportedwas planned. The fix is the same one experimental science uses: write down what would count as success before looking, and keep a record that shows the order things happened in.
    This is the protocol used for model experiments here. It adapts published work on adaptive preregistration for model-based research and on reporting deviations from a preregistered plan. It is written for simulations, analysis pipelines and research software, andwhoever foror thewhatever agentsis that now do much ofdoing the running.
    ## The five rules
    Each experiment gets its own branch and draft pull request. Tagging a milestone posts a receipt to the pull request with the commit and a checksum of the plan and the lockfile. The comment's timestamp comes from the hosting service, not from the machine that made the commit.
    The conversation that producedWhen the work is done with a coding assistant, the conversation is recorded too.: Each turn, theeach prompt and the agent's final reply areis posted to the same pull request and committed as a log beside the experiment, with secrets redacted. The diffs are already in the commits.
    ## Rules for thewhoever agentruns it
    The same rules apply to a person at the keyboard, a script, or a coding assistant.
    - Before running anything citable, confirm the plan is complete and tagged. If it is not, write it and stop for asomeone humanelse to review.
    - Never edit the plan after it is tagged. Changes go in the deviations table.
    - In every report, state how deeply each source was read: full text, abstract, background knowledge, or supplied by the user.
    - If a humananyone asks to skip a step, write the skip into the deviations table first, with the outcome-known field filled in honestly.
    ## How it reaches every project
    The protocol is maintained in one repository and installed into each project with a small command-line tool, `kit_ap`. It copies the protocol documents and tools into the project's `.agents/` folder and records the exact version in a lockfile. It also addswrites a short managed block ofthe rules tointo `AGENTS.md`, so coding assistants working in the repository read the same rules a person would.
    `AGENTS.md` is the file Codex, Copilot and Cursor read, and Claude Code reads it through an import in `CLAUDE.md`. Every agent sees the same rules whatever tool is open, including in cloud sandboxes that only have the repository.
    WhereSome a tool supports hooks,of the rules are backed by code rather than left to thegood agentintentions:
    - conversations are logged automatically;
    - Claude Code is told on every prompt whether the current experiment's plan is frozen;
    - an optional commit guard refuses outputs for an experiment that has no preregistration tag, and refuses any change to a plan after it is tagged;
    - where the tools support hooks, the current experiment's plan status is surfaced on every session, and assistant conversations are logged automatically;
    - at the start of a session, the project checks whether the protocol has changed upstream., The agentand reports it but does not update without permissionupdating on its own.
    ## Limits
    A local git date is not tamper-evident. The pull-request receipt is a third-party clock but not an archive. For outside credibility, the preregistration tag should also be deposited with an archive, and the plan should say which.
    Most of the protocol is instructionsrules anthat agentpeople isand tools are asked to follow. Only the commit guard and the hooks enforce anything, and the hooks only in tools that run hooksthem. The rest relies on the record making a violation visible afterwards.

    Commit cad5137 on GitHub →

  9. v1 2026-09-24 12:52 Publish: Adaptive preregistration when an agent runs the model Joshua Anderton with Claude Opus 5.5 first version

    v1: the original version, as first committed. Read it on GitHub →

    Commit 80f0041 on GitHub →