Work · Repository
scientific-research-scaffold
A protocol for studies and simulators across repositories, and a CLI that scaffolds and checks them.
- Python
- 1 post
- 3 merged PRs
Posts about it
README
A protocol for laying out research across repositories, and scaffold, a small CLI that sets up new
studies, simulators and tools in that shape and checks existing ones against it.
- The protocol:
PROTOCOL.md - Bringing an existing repo onto it:
ADOPTING.md
Three kinds of repo
| a study | a sim | a tool | |
|---|---|---|---|
| is | one research question, and everything used to pursue it | one simulator, under adaptive preregistration | one job done for other repos: an instrument, a recorder, a service, a library |
| holds | apps, sims still in development, datasets and calculators, each preregistered where it predicts | the model, its preregistrations, its assumptions | the tool, and the list of repos that depend on it and what they use |
| versions | apps/<slug>/v1, apps/<slug>/v2 folders, and tags |
model-vX.Y.Z tags, all in one repo |
vX.Y.Z tags, one version said the same everywhere |
| findings go to | the shared research corpus, referenced by claim ID | the same | none: a tool holds no research |
A sim starts as a folder inside the study that needs it, and moves to its own repo when a second study
uses it, it needs its own releases, or it is shared on its own. Repos are named for their subject, with no
lab- or sim- prefix: the kind goes in the manifest.
Studies and sims build on four other pieces, named in a profile so the protocol itself stays generic:
- a research corpus, the one shared record of claims and findings;
- a preregistration kit, adaptive-preregistration, which installs the predict-first protocol;
- a design system for the apps, polarize-ui in ours;
- a workbench of shared tools not yet released as libraries.
Quick start
git clone --recurse-submodules https://github.com/polarizetech/scientific-research-scaffold.git
git clone https://github.com/polarizetech/adaptive-preregistration.git # found automatically as a sibling
cd ~/code
export SCAFFOLD_PROFILE=example # or your own; there is no built-in default (see below)
scientific-research-scaffold/bin/scaffold new study reef-acoustics \
-q "Does the sound of a reef track its coral cover?" \
--hypothesis "A reef's soundscape predicts its ecological condition." --visibility private
cd reef-acoustics
../scientific-research-scaffold/bin/scaffold new app site-listener \
-q "Can a degraded site be heard?" --hypothesis-element H1
../scientific-research-scaffold/bin/scaffold new sim sound-propagation \
-q "How reef sound carries" --inside . --hypothesis-element H1
../scientific-research-scaffold/bin/scaffold check
new study renders the template, makes the first commit, installs the preregistration kit and prints
the gh repo create line. It never publishes anything. Visibility is yours to decide: without
--visibility public|private the manifest says undecided, and check fails until you record a decision.
| command | does |
|---|---|
scaffold new study <name> -q "..." --hypothesis "..." [--visibility public\|private] |
a new study repo with its required hypothesis preserved verbatim |
scaffold new sim <name> -q "..." [--inside STUDY] [--hypothesis-element H1] |
a new sim repo, or a sim folder linked into a study’s hypothesis map |
scaffold new tool <name> -j "..." [--inside STUDY\|WORKBENCH] |
a new tool repo, or a tool folder inside a study or workbench |
scaffold new workbench <name> -q "..." |
a workbench: general tools, sims and apps, and studies as folders (new study --inside) |
scaffold new app <slug> -q "..." --hypothesis-element H1 |
the app’s next version, apps/<slug>/vN/, registered in STUDY.toml and linked in HYPOTHESIS.toml |
scaffold new <app\|sim\|tool\|dataset\|calculator> ... --exploratory |
the same unit with no claim asked for: a build note in SCOPE.toml, and an exploratory label |
scaffold link-hypothesis <unit-path> --element H1 --role "..." |
associate an existing unit with accepted elements and write the same provenance into its local scope |
scaffold sync-hypothesis [PATH ...] |
detect changed element/evidence snapshots across affected studies and queue units for agent review without changing claims |
scaffold promote-sim sims/<slug> |
split a sim out of a study into its own repo, keeping its history |
scaffold check [PATH] |
check a repo against the protocol; nonzero exit on failure, for CI |
scaffold status [PATH ...] |
one line per repo: kind, stage, scaffold release, CI ref, check result, pending update |
scaffold update [PATH] [--apply] [--adopt FILE] |
bring the files the scaffold owns up to this release; a dry run unless --apply |
scaffold version [PATH] |
whether a repo follows the current scaffold, and the steps to upgrade it (sessions run this themselves) |
scaffold usage [PATH ...] [--days N] [--all] |
tokens and API-equivalent cost per repo from Claude Code and Codex logs, and Codex’s plan limits |
scaffold profiles |
list profiles |
Agents
Every repo gets specialist role briefs in .claude/agents/: a scout, a computational engineer, a designer, a
frontend developer, a researcher, an analyst and a science writer. A sim, which has no UI, gets no designer
or frontend developer. Claude Code discovers them as subagents; ChatGPT, Codex and other repository-aware
assistants read the same briefs through AGENTS.md and can apply them in the current session when subagents
are unavailable. Each agent starts on a cheap model and is escalated on a signal, and agents pass short handoffs, not transcripts
(agents/COORDINATION.md). scaffold usage shows what each repo actually costs.
The agents live in their own repo,
scientific-research-agents, included here as a
submodule pinned to a tag: clone with --recurse-submodules, or run git submodule update --init.
skills/new-study/ is a portable skill for creating and adopting studies with this scaffold. Install the
whole repository as a skills-only plugin in ChatGPT or Codex, or copy the skill to the host’s skill folder:
- Claude Code:
~/.claude/skills/new-study/or.claude/skills/new-study/. - Codex:
~/.codex/skills/new-study/or.codex/skills/new-study/.
Plain ChatGPT does not operate on an unconnected local checkout. Use the plugin in a workspace with local repository access, or use Codex for the filesystem and shell steps.
Keeping repos current
scaffold status ~/code/* surveys every repo at once. scaffold update brings one repo’s
scaffold-owned files (the CI workflow, the Makefile, shared/workbench.py, the agents) up to this release,
and never touches anything else. It replaces a file only when nobody has edited it since the scaffold
wrote it, which it knows from .scaffold.lock, or by matching the file against what each earlier
release would have written. An edited CI workflow gets only its scaffold ref: moved. Any other
edited file is kept, with the difference shown; --adopt FILE replaces it anyway. It prints a dry
run first, writes nothing without --apply, refuses to overwrite uncommitted work, and never commits.
Every release says how to upgrade to it in UPGRADING.md, and every repo notices: a session hook
runs scaffold version when a session starts and, if the repo is behind, prints the steps and offers to do them.
Requirements: Python 3.9+ and git. No other dependencies.
Using it for your own work
The protocol is written to be forked. Copy profiles/example.toml, name your own
corpus, preregistration kit, design system and workbench (any can be left empty), and pass
--profile <name> (or set SCAFFOLD_PROFILE). There is no default profile, so nothing of ours ends up in your
repos by accident. profiles/polarizetech.toml is ours, as a worked example.
Licence
Code is MIT; the protocol and documentation are CC BY 4.0. See LICENSE. To cite it, see
CITATION.cff.
Changelog
Pin agents v0.4.0 (stricter designer)
Bumps the agents submodule to v0.4.0: the designer brief gains an Originality section and an Audit mode.
🤖 Generated with Claude Code
Apps are laid out as apps//vN, each app numbered from v1
An app’s versions now sit together: apps/entropy-media/v1/, apps/entropy-media/v2/. Before, a study had one number sequence and folders named apps/vN-<slug>/, which put unrelated apps side by side in number order and split one app’s versions apart.
What changes
scaffold new app <slug>writesapps/<slug>/vN/. A new slug starts atv1; a slug that exists gets its next version.checklooks for an[[apps]]entry with nopathatapps/<slug>/<version>/, then at the earlierapps/<version>-<slug>/. Repos on the earlier layout pass unchanged (checked againstcochlear-mirrorandtinnitus).- The app template’s
serve.pylooks one folder further up for the design system. PROTOCOL.md§ 1, 2, 3, 5, 6,README.md,ADOPTING.mdand the study template’sapps/README.mddescribe the new layout.
Not in this PR
- No release: the changelog entry is under “Unreleased”, and
SCAFFOLD_REF,UPGRADING.mdand the generated CI ref are untouched. - No migration command. Existing repos keep their folders;
world-seedingalready uses the new layout throughpathentries. - The preregistration kit reads the same default in polarizetech/adaptive-preregistration (its own PR). Until both are released, a repo on the new layout needs
pathin each entry for the kit to find its apps.
Tests: 73 pass (2 new: per-app version numbers, and the earlier layout passing check).
🤖 Generated with Claude Code
Release v0.8.1: generated CI can install private dependencies
Why
A repo that pins the organisation’s private repos by git URL in pyproject.toml fails at uv sync in GitHub Actions (fatal: could not read Username for 'https://github.com': terminal prompts disabled), because the default GITHUB_TOKEN reads only its own repo. make check is then skipped, so the scaffold check never runs. This was worked around by hand in polarizetech/matched-entropy-demodulation#11.
What
- Templates (study, sim, tool, workbench): the workflow reads an optional
PRIVATE_DEPS_TOKENsecret and, when it is set, rewriteshttps://github.com/{{org}}/to use it. The org comes from the profile, not hard-coded. - Install step: always runs
uv sync --frozen || uv sync. A failure becomes a::warning::only when the token is unset andpyproject.tomloruv.lockmentionsgithub.com/<org>/; any other failed install still fails the job. - Docs: the secret is described in PROTOCOL.md § 4; UPGRADING.md and CHANGELOG.md have their v0.8.1 sections.
- Version: 0.8.1 in
bin/scaffold,CITATION.cffand bothplugin.jsonfiles.
Difference from the hand fix
matched-entropy-demodulation#11 skips uv sync whenever the token is unset. This does not, so a repo with only public dependencies and no secret keeps installing as before.
Upgrading from v0.8.0
- An unedited v0.8.0 workflow is replaced by
scaffold update(by lock hash, or without a lock by matching what v0.8.0 wrote, which needs the scaffold’s git history and tag). - An edited workflow keeps its edits and only the
ref:moves;--adopt .github/workflows/check.ymlreplaces it.
Tests
python3 -m unittest discover tests: 67 run, 1 skipped (needs Python 3.11+), all pass locally. New tests cover the rendered org per profile, the install step’s three outcomes with a stand-in uv, and the v0.8.0 upgrade with and without a lock. One existing test changed: it asserted no {{ anywhere in a new repo, which ${{ secrets.… }} trips; it now looks for unfilled {{placeholder}} patterns.
The generated workflow has not yet run in GitHub Actions.
Not in this PR
Tagging v0.8.1 after merge. With private pins and no secret, make check still fails afterwards in a repo that has tests/, since uv run pytest needs the dependencies.
🤖 Generated with Claude Code