Polarize
GitHub Get in touch

Work · Repository

scientific-research-scaffold

A protocol for studies and simulators across repositories, and a CLI that scaffolds and checks them.

  • Python
  • 1 post
  • 3 merged PRs

View on GitHub → Changelog ↓

Posts about it

README

A protocol for laying out research across repositories, and scaffold, a small CLI that sets up new studies, simulators and tools in that shape and checks existing ones against it.

Three kinds of repo

  a study a sim a tool
is one research question, and everything used to pursue it one simulator, under adaptive preregistration one job done for other repos: an instrument, a recorder, a service, a library
holds apps, sims still in development, datasets and calculators, each preregistered where it predicts the model, its preregistrations, its assumptions the tool, and the list of repos that depend on it and what they use
versions apps/<slug>/v1, apps/<slug>/v2 folders, and tags model-vX.Y.Z tags, all in one repo vX.Y.Z tags, one version said the same everywhere
findings go to the shared research corpus, referenced by claim ID the same none: a tool holds no research

A sim starts as a folder inside the study that needs it, and moves to its own repo when a second study uses it, it needs its own releases, or it is shared on its own. Repos are named for their subject, with no lab- or sim- prefix: the kind goes in the manifest.

Studies and sims build on four other pieces, named in a profile so the protocol itself stays generic:

  • a research corpus, the one shared record of claims and findings;
  • a preregistration kit, adaptive-preregistration, which installs the predict-first protocol;
  • a design system for the apps, polarize-ui in ours;
  • a workbench of shared tools not yet released as libraries.

Quick start

git clone --recurse-submodules https://github.com/polarizetech/scientific-research-scaffold.git
git clone https://github.com/polarizetech/adaptive-preregistration.git   # found automatically as a sibling
cd ~/code
export SCAFFOLD_PROFILE=example   # or your own; there is no built-in default (see below)

scientific-research-scaffold/bin/scaffold new study reef-acoustics \
  -q "Does the sound of a reef track its coral cover?" \
  --hypothesis "A reef's soundscape predicts its ecological condition." --visibility private
cd reef-acoustics
../scientific-research-scaffold/bin/scaffold new app site-listener \
  -q "Can a degraded site be heard?" --hypothesis-element H1
../scientific-research-scaffold/bin/scaffold new sim sound-propagation \
  -q "How reef sound carries" --inside . --hypothesis-element H1
../scientific-research-scaffold/bin/scaffold check

new study renders the template, makes the first commit, installs the preregistration kit and prints the gh repo create line. It never publishes anything. Visibility is yours to decide: without --visibility public|private the manifest says undecided, and check fails until you record a decision.

command does
scaffold new study <name> -q "..." --hypothesis "..." [--visibility public\|private] a new study repo with its required hypothesis preserved verbatim
scaffold new sim <name> -q "..." [--inside STUDY] [--hypothesis-element H1] a new sim repo, or a sim folder linked into a study’s hypothesis map
scaffold new tool <name> -j "..." [--inside STUDY\|WORKBENCH] a new tool repo, or a tool folder inside a study or workbench
scaffold new workbench <name> -q "..." a workbench: general tools, sims and apps, and studies as folders (new study --inside)
scaffold new app <slug> -q "..." --hypothesis-element H1 the app’s next version, apps/<slug>/vN/, registered in STUDY.toml and linked in HYPOTHESIS.toml
scaffold new <app\|sim\|tool\|dataset\|calculator> ... --exploratory the same unit with no claim asked for: a build note in SCOPE.toml, and an exploratory label
scaffold link-hypothesis <unit-path> --element H1 --role "..." associate an existing unit with accepted elements and write the same provenance into its local scope
scaffold sync-hypothesis [PATH ...] detect changed element/evidence snapshots across affected studies and queue units for agent review without changing claims
scaffold promote-sim sims/<slug> split a sim out of a study into its own repo, keeping its history
scaffold check [PATH] check a repo against the protocol; nonzero exit on failure, for CI
scaffold status [PATH ...] one line per repo: kind, stage, scaffold release, CI ref, check result, pending update
scaffold update [PATH] [--apply] [--adopt FILE] bring the files the scaffold owns up to this release; a dry run unless --apply
scaffold version [PATH] whether a repo follows the current scaffold, and the steps to upgrade it (sessions run this themselves)
scaffold usage [PATH ...] [--days N] [--all] tokens and API-equivalent cost per repo from Claude Code and Codex logs, and Codex’s plan limits
scaffold profiles list profiles

Agents

Every repo gets specialist role briefs in .claude/agents/: a scout, a computational engineer, a designer, a frontend developer, a researcher, an analyst and a science writer. A sim, which has no UI, gets no designer or frontend developer. Claude Code discovers them as subagents; ChatGPT, Codex and other repository-aware assistants read the same briefs through AGENTS.md and can apply them in the current session when subagents are unavailable. Each agent starts on a cheap model and is escalated on a signal, and agents pass short handoffs, not transcripts (agents/COORDINATION.md). scaffold usage shows what each repo actually costs. The agents live in their own repo, scientific-research-agents, included here as a submodule pinned to a tag: clone with --recurse-submodules, or run git submodule update --init.

skills/new-study/ is a portable skill for creating and adopting studies with this scaffold. Install the whole repository as a skills-only plugin in ChatGPT or Codex, or copy the skill to the host’s skill folder:

  • Claude Code: ~/.claude/skills/new-study/ or .claude/skills/new-study/.
  • Codex: ~/.codex/skills/new-study/ or .codex/skills/new-study/.

Plain ChatGPT does not operate on an unconnected local checkout. Use the plugin in a workspace with local repository access, or use Codex for the filesystem and shell steps.

Keeping repos current

scaffold status ~/code/* surveys every repo at once. scaffold update brings one repo’s scaffold-owned files (the CI workflow, the Makefile, shared/workbench.py, the agents) up to this release, and never touches anything else. It replaces a file only when nobody has edited it since the scaffold wrote it, which it knows from .scaffold.lock, or by matching the file against what each earlier release would have written. An edited CI workflow gets only its scaffold ref: moved. Any other edited file is kept, with the difference shown; --adopt FILE replaces it anyway. It prints a dry run first, writes nothing without --apply, refuses to overwrite uncommitted work, and never commits.

Every release says how to upgrade to it in UPGRADING.md, and every repo notices: a session hook runs scaffold version when a session starts and, if the repo is behind, prints the steps and offers to do them.

Requirements: Python 3.9+ and git. No other dependencies.

Using it for your own work

The protocol is written to be forked. Copy profiles/example.toml, name your own corpus, preregistration kit, design system and workbench (any can be left empty), and pass --profile <name> (or set SCAFFOLD_PROFILE). There is no default profile, so nothing of ours ends up in your repos by accident. profiles/polarizetech.toml is ours, as a worked example.

Licence

Code is MIT; the protocol and documentation are CC BY 4.0. See LICENSE. To cite it, see CITATION.cff.

Changelog

2026-10-18 #3

Pin agents v0.4.0 (stricter designer)

Bumps the agents submodule to v0.4.0: the designer brief gains an Originality section and an Audit mode.

🤖 Generated with Claude Code

2026-10-13 #2

Apps are laid out as apps//vN, each app numbered from v1

An app’s versions now sit together: apps/entropy-media/v1/, apps/entropy-media/v2/. Before, a study had one number sequence and folders named apps/vN-<slug>/, which put unrelated apps side by side in number order and split one app’s versions apart.

What changes

  • scaffold new app <slug> writes apps/<slug>/vN/. A new slug starts at v1; a slug that exists gets its next version.
  • check looks for an [[apps]] entry with no path at apps/<slug>/<version>/, then at the earlier apps/<version>-<slug>/. Repos on the earlier layout pass unchanged (checked against cochlear-mirror and tinnitus).
  • The app template’s serve.py looks one folder further up for the design system.
  • PROTOCOL.md § 1, 2, 3, 5, 6, README.md, ADOPTING.md and the study template’s apps/README.md describe the new layout.

Not in this PR

  • No release: the changelog entry is under “Unreleased”, and SCAFFOLD_REF, UPGRADING.md and the generated CI ref are untouched.
  • No migration command. Existing repos keep their folders; world-seeding already uses the new layout through path entries.
  • The preregistration kit reads the same default in polarizetech/adaptive-preregistration (its own PR). Until both are released, a repo on the new layout needs path in each entry for the kit to find its apps.

Tests: 73 pass (2 new: per-app version numbers, and the earlier layout passing check).

🤖 Generated with Claude Code

2026-10-06 #1

Release v0.8.1: generated CI can install private dependencies

Why

A repo that pins the organisation’s private repos by git URL in pyproject.toml fails at uv sync in GitHub Actions (fatal: could not read Username for 'https://github.com': terminal prompts disabled), because the default GITHUB_TOKEN reads only its own repo. make check is then skipped, so the scaffold check never runs. This was worked around by hand in polarizetech/matched-entropy-demodulation#11.

What

  • Templates (study, sim, tool, workbench): the workflow reads an optional PRIVATE_DEPS_TOKEN secret and, when it is set, rewrites https://github.com/{{org}}/ to use it. The org comes from the profile, not hard-coded.
  • Install step: always runs uv sync --frozen || uv sync. A failure becomes a ::warning:: only when the token is unset and pyproject.toml or uv.lock mentions github.com/<org>/; any other failed install still fails the job.
  • Docs: the secret is described in PROTOCOL.md § 4; UPGRADING.md and CHANGELOG.md have their v0.8.1 sections.
  • Version: 0.8.1 in bin/scaffold, CITATION.cff and both plugin.json files.

Difference from the hand fix

matched-entropy-demodulation#11 skips uv sync whenever the token is unset. This does not, so a repo with only public dependencies and no secret keeps installing as before.

Upgrading from v0.8.0

  • An unedited v0.8.0 workflow is replaced by scaffold update (by lock hash, or without a lock by matching what v0.8.0 wrote, which needs the scaffold’s git history and tag).
  • An edited workflow keeps its edits and only the ref: moves; --adopt .github/workflows/check.yml replaces it.

Tests

python3 -m unittest discover tests: 67 run, 1 skipped (needs Python 3.11+), all pass locally. New tests cover the rendered org per profile, the install step’s three outcomes with a stand-in uv, and the v0.8.0 upgrade with and without a lock. One existing test changed: it asserted no {{ anywhere in a new repo, which ${{ secrets.… }} trips; it now looks for unfilled {{placeholder}} patterns.

The generated workflow has not yet run in GitHub Actions.

Not in this PR

Tagging v0.8.1 after merge. With private pins and no secret, make check still fails afterwards in a repo that has tests/, since uv run pytest needs the dependencies.

🤖 Generated with Claude Code