What changed in the code.

Summary

Makes scite opt-in. Its subscription allows 250 MCP calls a month, and one day in the default search set spent all of it.

After #7, every paper-fetch search asked scite, and a profile search asked it up to three times (once per query variant). With the research-rag gateway and about a dozen sessions searching, the month’s allowance was gone within a day. scite now answers “You have reached your monthly MCP usage limit (250 calls)”.

What changes

• Default set: scite is no longer in DEFAULT_SEARCH. An ordinary search doesn’t touch it.
• Opting in: a search asks scite on top of the usual providers with also=["scite"] in Python, --scite on the CLI, or scite: true on the MCP search tool. Naming it in PAPER_FETCH_SEARCH_PROVIDERS or providers= still works.
• One call per search: Provider.metered is new and is True for scite. A metered provider is asked the query as written, once, and never a profile’s variants.
• Session reuse: the opted-in provider is built once per Library, so scite’s MCP session is reused instead of opened per search. One thread uses that session at a time.
• Docs: README and CLAUDE.md record the limit and how to opt in.

Test plan

• uv run pytest: 329 passed, 95% coverage. New tests cover:
  • scite is asked only when told to;
  • also doesn’t ask a provider twice and reuses the session;
  • a metered provider is skipped for profile variants;
  • the CLI flag and the MCP parameter pass the opt-in through.
• ruff check, ruff format --check and pyright are clean.
• Live: a default paper-fetch search reports the nine usual providers and no scite.
• Live: --scite asks scite by name. It currently returns scite’s monthly-limit message, reported as unavailable.
• A successful live scite search through the opt-in path. This can’t run until the allowance resets.

🤖 Generated with Claude Code

Follow-up to #8. An MCP server runs every tool call on a worker thread, so one Library is used from several threads at once. #8 locked the passage index’s connection. This PR covers the rest of the state those threads share.

• Catalogue:
  • One lock guards the in-memory dictionary and its save.
  • Readers that walk the catalogue take a snapshot (_rows()). Writers go through _record / _forget.
  • A save now updates the dictionary in place without emptying it. Before, a lookup from another thread mid-save could find a held paper missing and fetch it again.
  • rebuild_index builds the new catalogue aside and swaps it in whole.
• Search memory and collections: each read-modify-write holds a lock, so no thread’s row or member is lost.
• Lazily built parts (profiles, memory, passage index, reranker): built once.
• Provider request spacing: two threads could both pass the wait at once and break a rate limit such as PLOS’s 10 per minute. The wait is now taken under a per-provider lock.

No lock is held across a fetch or a search, so slow network work does not block other calls. The locks are per process; the cross-process behaviour is unchanged.

Tested:

• tests/test_threads.py runs each part from 16 threads. All five tests fail on main and pass here.
• The thread tests passed 25 runs in a row.
• All 324 tests pass, and ruff and pyright are clean.

🤖 Generated with Claude Code

What failed: retrieve failed intermittently in service. The gateway log shows 13 failed calls. Eleven are sqlite3.InterfaceError: bad parameter or other API misuse, and one is passage text read back as undecodable bytes.

Cause: the MCP library runs every sync tool on a worker thread (anyio.to_thread.run_sync), so overlapping retrieve calls used the passage index’s single SQLite connection at the same time. The index was written as if calls were serial.

Fix: PassageIndex holds an RLock around every use of its connection, including reads, writes and the cached vector matrix. Embedding, the slow part of indexing, stays outside the lock, so indexing a paper does not block retrieval for seconds.

Tested:

• A new test runs 200 mixed search / passages / paper / stats / add calls across 16 threads on one index. It fails on main and passes here.
• All 319 tests pass, and ruff and pyright are clean.

Not covered: Library’s other shared state (the in-memory catalogue) is also reached from several threads. No failure from it appears in the logs, so this PR leaves it alone.

🤖 Generated with Claude Code

Why

A repo that pins the organisation’s private repos by git URL in pyproject.toml fails at uv sync in GitHub Actions (fatal: could not read Username for 'https://github.com': terminal prompts disabled), because the default GITHUB_TOKEN reads only its own repo. make check is then skipped, so the scaffold check never runs. This was worked around by hand in polarizetech/matched-entropy-demodulation#11.

What

• Templates (study, sim, tool, workbench): the workflow reads an optional PRIVATE_DEPS_TOKEN secret and, when it is set, rewrites https://github.com/{{org}}/ to use it. The org comes from the profile, not hard-coded.
• Install step: always runs uv sync --frozen || uv sync. A failure becomes a ::warning:: only when the token is unset and pyproject.toml or uv.lock mentions github.com/<org>/; any other failed install still fails the job.
• Docs: the secret is described in PROTOCOL.md § 4; UPGRADING.md and CHANGELOG.md have their v0.8.1 sections.
• Version: 0.8.1 in bin/scaffold, CITATION.cff and both plugin.json files.

Difference from the hand fix

matched-entropy-demodulation#11 skips uv sync whenever the token is unset. This does not, so a repo with only public dependencies and no secret keeps installing as before.

Upgrading from v0.8.0

• An unedited v0.8.0 workflow is replaced by scaffold update (by lock hash, or without a lock by matching what v0.8.0 wrote, which needs the scaffold’s git history and tag).
• An edited workflow keeps its edits and only the ref: moves; --adopt .github/workflows/check.yml replaces it.

Tests

python3 -m unittest discover tests: 67 run, 1 skipped (needs Python 3.11+), all pass locally. New tests cover the rendered org per profile, the install step’s three outcomes with a stand-in uv, and the v0.8.0 upgrade with and without a lock. One existing test changed: it asserted no {{ anywhere in a new repo, which ${{ secrets.… }} trips; it now looks for unfilled {{placeholder}} patterns.

The generated workflow has not yet run in GitHub Actions.

Not in this PR

Tagging v0.8.1 after merge. With private pins and no secret, make check still fails afterwards in a repo that has tests/, since uv run pytest needs the dependencies.

🤖 Generated with Claude Code

Summary

Adds scite as a search provider alongside OpenAlex and the others. It uses your scite subscription and needs no API key.

scite’s REST search (/api_partner/search) returns 401 without a partner token, which comes from their sales team. Its MCP server (https://api.scite.ai/mcp) accepts the subscriber’s own sign-in instead: OAuth 2.1 with dynamic client registration and PKCE. So the provider is a small MCP client written against Http. It sends initialize, then tools/call search_literature.

What changes

• Sign-in: paper-fetch scite-login opens the browser once and stores the sign-in in ~/.config/paper-fetch/scite-oauth.json (mode 0600). It is never written to the shared store. scite-logout removes it.
• Token renewal: the access token is renewed automatically. scite rotates the refresh token on every use, so the file is re-read before a refresh and once more after a refused one.
• Provider: providers/scite.py is search-only, like doaj. It is in the default search set and reported skipped until someone signs in. Its results are not cached.
• What a hit keeps: DOI, title, year, three authors, scite’s open-access flag and status, the citation tally and editorial notices.
• What a hit drops: scite’s access links (they carry the account’s email address), abstracts and citation statements.
• Merged search results: a hit that scite found carries a scite key with the tally, oa_status and editorial_notices.
• HTTP layer: Http.post is new. It returns a 4xx response as an answer, the way get does. http.py stays the only module that calls urlopen.
• Docs: README, CLAUDE.md and INTEGRATION.md are updated.

Design notes

• No mcp SDK in the core. The core stays standard-library plus pypdf, and all outbound requests still go through Http. The one new socket is the loopback listener that receives the browser redirect during login.
• oa_only: the tool has no open-access filter. The provider asks for three times limit (capped at 50, because every hit carries a citation list) and drops what scite marks closed.
• Failures: an expired sign-in, a 429, a tool error or an unreachable server is reported as unavailable with what to do, never as zero results.

Test plan

• uv run pytest: 318 passed, 95% coverage (29 new tests in tests/test_scite.py, against a fake scite server).
• ruff check, ruff format --check and pyright are clean.
• Live sign-in through the browser on 2026-10-05; the stored file is mode 0600.
• Live search: 5 hits in 3.7 s (3 POSTs on a new session), with nothing from the access links kept.
• Live oa_only search: every hit returned is open.
• Live token refresh: forced by expiring the token locally. The access token changed and the refresh token rotated.
• Live federated search: paper-fetch search … --providers scite,openalex,europepmc merged hits across the three providers.

🤖 Generated with Claude Code

dataset-fetch now has recommend(topic) (polarizetech/dataset-fetch#12). Choosing catalogues, phrasing queries and ranking datasets now live there, the way paper search lives in paper-fetch. This switches the datasets companion to that one call.

• Calls: one recommend call per planner query, at most two.
• Merging: cards are merged by reference, keeping the better score. Records with verified data files come first, then each by dataset-fetch’s own score.
• Printing: the answer shows the tool’s own fields: catalogue and kind, licence, files, subjects, modalities, creators, DOI, a pinned reference, and the tool’s reason for listing it. Records that hold only documents, and lower-ranked records, are counted rather than listed.
• Failures: a query that recommend refuses doesn’t discard the other query’s results. A catalogue that was silent on every query is named in Limits.
• Removed: the companion’s own loop over catalogues, the Zenodo query rewrite, the card reading and the document filter.

Tested:

• 238 tests pass, and ruff and pyright are clean.
• Live: one query, about 19 s.
  • dataset-fetch asked OpenNeuro, PhysioNet, Zenodo and OSF, and skipped the other ten with reasons.
  • Every listed record held data, including PhysioNet blood-pressure databases and a structured-exercise dataset.
  • The old loop listed none of six.

Needs a dataset-fetch installed with recommend.

🤖 Generated with Claude Code

The bug

quality.spectrum() promised to summarise Goertzel bins into display buckets, but evaluated only every step-th DFT bin and skipped the rest. On a record longer than a couple of seconds a narrow line fell between the evaluated bins and disappeared. A 260-count 10 Hz sine in noise over 20 s drew its tallest point at 35.65 Hz, about 1.7 counts. The Monitor tab’s spectrum (POST /spectrum, up to 20 s every ~500 ms) was affected the same way.

The fix

• Every transform bin in the range is evaluated, and each bucket reports the maximum amplitude of the bins it covers. A maximum keeps a narrow line at its own amplitude however long the record is; a mean of power would dilute it. The price, stated in the docstring: the noise floor of the display rises slowly with record length, and it is not a power density.
• Cost: one stdlib radix-2 FFT of the Hann-windowed record, zero-padded to the next power of two at or above twice its length, over at most the most recent 16,384 samples (65 s at 250 Hz). The padding bounds the scalloping loss at 4 %.
• freqs are now bucket centres. resolution_hz keeps its meaning (rate / samples used). New fields: bucket_hz, bin_hz, samples_used, summary. The web client needs no change.

Checks

• apps/server/dev_check.py: a 10 Hz sine in noise at 2 s, 20 s and 60 s must be the tallest point, within one bucket of 10 Hz and within 10 % of the input amplitude; plus a check on the sample cap.
• python3 scripts/check.py: all seven suites pass. check_docs.py run against the blog post: in step. ruff clean.

Versioning

No version bump and no model-v* tag. quality.py is in the measuring chain, but spectrum() feeds only the drawing: nothing it returns is stored in a session or an export, and assess() and bands() are unchanged. The CHANGELOG entry says so. If this should be 0.2.1 instead, that is a follow-up commit here.

🤖 Generated with Claude Code

What this adds

A generated docs site for this repository, built by automatic-documentation-site. Nothing on it is written by hand: a page is fixed by fixing the Markdown file, docstring or story it names.

No code under apps/ changes.

What the site contains (22 pages)

• Guides: the README (with Safety and the board photos), then the hardware, recording, standards, versioning, bridge, workbench and project documents.
• API reference: apps/server and apps/web Python, and the exported browser modules. 85 public symbols, read without importing or running anything.
• Stories, on synthetic or modelled inputs only, never a recording of a person:
  • what the front end can carry (rig.carries): alpha reachable, brainstem responses unreachable
  • which readouts fit in its band (readouts.gate)
  • the pre-flight verdict on a good, a flat and a railed channel (quality.assess)
  • the delivered sample rate (timebase.RateEstimator recovers a simulated 248.3 Hz)

One story records a check that does not hold

quality.spectrum of a clean 10 Hz sine has no peak at 10 Hz on a 20 s record. Its docstring says it averages Goertzel bins into display buckets; the code evaluates every step-th bin, so a narrow line between them is missed. The story shows the check as it came out (“absent”) and was not adjusted to pass. The fix belongs in a separate change.

Checked

• docsite build . --refresh --strict --publish pages with the released tool (v0.2.1) and polarize-ui v0.5.15: 22 pages, 4/4 stories run.
• python3 scripts/check.py: every suite passes.
• The built site opens from a folder (file://) with plots, search and badges working.

After merging

The workflow runs on main and deploys to https://polarizetech.github.io/olimex-shield/ (GitHub Pages is enabled, source: GitHub Actions).

Based on main. The extension catalogue on the tool-only work is not on main yet; when it lands it is one [catalogue] block in docs-site.toml.

🤖 Generated with Claude Code

Found in a live run of retrieval-augmented-generation: a bibliography entry from a PDF was retrieved as a passage. The chunker only cut at a paragraph that was exactly “References”. PDF text often has the heading on a line inside a paragraph, or loses it altogether.

Detection (references_start)

• A heading on a line of its own (“References”, “6. References”, “Literature Cited”), past the first third of the text.
• Or, with no heading: a run of at least 8 consecutively numbered entries starting at 1 that begins in the second half of the text, reaches its last tenth, and mostly carries publication years.
• Numbered lists in the body are kept. That covers lists that are short, early, have no years, or do not start at 1.

Measured on 80 held papers: 11 of 12 PDFs are now cut (none were before), every cut I read lands at the start of the list, and the 68 JATS and TEI texts are unaffected. An unnumbered author-year list with no heading is still not recognised.

Existing indexes

• The chunker has a version (CHUNKER = 2), kept in a new chunked table next to the indexed_at of the row it describes.
• The papers table is unchanged, so servers still running older code and sharing the file keep working. A paper one of them re-indexes counts as stale again.
• has() is false for a stale paper, so it is re-chunked the next time it is indexed. Passages whose text did not change keep their vectors, so only new text is embedded.
• status reports stale.

Tested: 287 tests pass, including the detector’s positive and negative cases, vector reuse, an index from before versions existed, and a row rewritten by older code. ruff and pyright are clean.

🤖 Generated with Claude Code

Problems

1. Works the library creates itself (`pmcid-PMC5519140`, `doi-10.1214%2F12-aoas565supp`) were rejected by `index`, `collect` and the MCP `index`/`collect` tools with “unrecognised identifier”. One bad id also failed a whole batch.
2. `paper-fetch index` with no arguments aborted on the first unparseable record, leaving later works unindexed.

Changes

• `ids.normalize` parses `doi-`/`pmcid-`/`pmid-`/`arxiv-` work ids, including percent-encoded DOIs. Behaviour for ordinary DOI / OpenAlex / PMID / PMCID inputs is unchanged and junk is still refused.
• `index_works` reads text from the catalogue row (no re-parse) and skips and reports unreadable records under `skipped`; the CLI prints each with its reason and a count.
• `collect` returns a `rejected` list instead of failing the batch.
• When OpenAlex has no record, `fetch` fills title, year, authors and DOI from Europe PMC (never fatal).
• Existing library records are not rewritten; already-stored metadata-less works are only enriched on a new `fetch`.

Tests

`tests/test_own_work_ids.py`: 12 of its 19 cases fail on main and all pass here; full suite 278 passed, ruff and pyright clean.

🤖 Generated with Claude Code