What changed in the code.

What

mountTrajectory3D(host, options) draws a path through three dimensions on a plain 2-D canvas, with a marker at the current time and the last trail seconds drawn heavier. It returns {update, setTime, show, resetView, destroy}. Zero-build, no dependencies, same shape as cellfield.js (a mount* function plus a .d.ts).

• trajectory3d.js + trajectory3d.d.ts at the repo root
• package.json: shipped in files, new entry point ./trajectory3d.js
• dev_check.py: check_trajectory3d(); scripts/api_surface.py tracks the export and the handle methods, so a later rename forces a breaking bump
• Story: Science / 3-D views / Path with a time marker (zero-build, synthetic)
• Docs: INTEGRATION.md § 5 (and the allow-list line in § 1), a row in README.md

Versioning

Additive, so a patch. A new entry point and nothing existing changes. The React Trajectory3D (the hodogram on Scene3D) is a separate view and is untouched; no React wrapper was added for this module, to avoid a name collision.

Public-repo rule

• Method: a 3-D state-space path / phase portrait with a playhead. Standard.
• Data: the story integrates the Lorenz system (Lorenz 1963) in the story itself. Synthetic, and labelled so.
• Findings: none shown or stated.

Differences from the private copy it was ported from

• Axes are right-handed and positive pitch looks down, as in src/science/space (the original was left-handed, seen from below). Paths appear mirrored relative to the original.
• On a drag the near side follows the pointer (direction reversed).
• A two-finger touch pinch zooms. Before, only Ctrl/Cmd-scroll did.
• Arrow keys turn and +/- zoom when the canvas has focus.
• Redraws on a theme change; falls back to --chart-1 / --chart-4 on the React theme.

Checked

• python3 dev_check.py: 429/429
• npx tsc -p .: clean
• python3 scripts/api_surface.py --removed v0.5.16: nothing removed
• In a browser: a zero-build page on design.css and the Storybook story, light and dark, and at 375 px (no sideways scroll). Turn, pinch, Ctrl-scroll zoom, keyboard, reset, destroy and live theme switch work; a plain scroll is left to the page. Pinch was tested with synthetic pointer events, not on a real phone.

Note for consumers

The canvas takes every touch that starts on it (touch-action: none), so a page needs scroll room beside or below it on a phone. Documented in INTEGRATION.md.

🤖 Generated with Claude Code

Adds a second companion tool, built the same way as datasets. It is off until enabled, and is asked only when the planner judges that the question is about a device, a method or another invention.

A companion can now be a command

• patent-fetch ships a command line and no MCP server, and offers patent-fetch --json search for gateways.
• upstream.Command runs a configured command and reads its standard output as JSON.
• The command is started with an argument list, never through a shell, so a query a model wrote is always one argument.
• It reports a missing command or a timeout as unavailable, and a non-zero exit with the last line of stderr.
• Companion.transport (mcp or command) picks the client.

The patents companion

• Runs one search per planner query and merges hits by publication number.
• patent-fetch does not rank, so the hits are ordered by the paper library’s relevance of each title to the question.
• Prints the services’ own records under “Existing patents on this”: number, year, applicants or inventors, classes, status in the service’s words, and a link.
• Fetches nothing, and never becomes evidence for a claim.
• The section states that it is not a legal opinion and that finding none is not evidence that none exists.
• Limits name every service that had no account or did not answer. They also say when only Europe PMC’s archive answered (life-science patents, none after 2012), so a search that could not see recent patents does not read as “none exist”.

Config (config/mcp-gateway.example.json): "patents": {"command": "patent-fetch", "args": ["--json"], "enabled": false}.

Tested:

• 249 tests pass, and ruff and pyright are clean.
• New tests cover merging and ordering, each limits case, and the command transport: a shell-hostile query passed as one argument, a non-zero exit, non-JSON output, a timeout, and a command that is not installed.
• New tests also cover a full run that requests patents, and a run where the tool is missing and the run continues.
• A live check against the real command returned 16 records for two queries, listed six, and gave the correct limits.

🤖 Generated with Claude Code

Summary

The last resort after the scholarly providers is a web search. Until now that needed a hosted SearXNG instance; without one, the fallback was skipped.

paper-fetch runs no language model, but its caller is usually a model with its own web search. This PR hands the last step to the caller, so nothing needs hosting.

• ask_the_web in search results. When the fallback is wanted (no open-access hit) and the library’s own web search did not answer, the result carries ask_the_web with why, the query, what happened to the library’s own web search, and how to proceed. “Did not answer” means SEARXNG_URL is unset or SearXNG is down.
• leads, available as Library.leads, the MCP leads tool and paper-fetch leads <url> .... It takes the URLs the caller found. It reads a DOI, PMID, PMCID or arXiv id out of each, using the web provider’s own parser, marks what is already held, and returns what to fetch. It makes no network calls. A URL with no identifier is returned under no_identifier rather than dropped.
• MCP instructions tell the model what to do when it sees ask_the_web.

The rule is unchanged: nothing is downloaded from a web page. An identifier goes through the open-access providers, which decide whether a copy may be kept.

SearXNG still works when configured, and then nothing is asked of the caller. web_fallback=False and PAPER_FETCH_WEB_FALLBACK=0 turn off the request along with the fallback.

Test plan

• uv run pytest: 375 passed, 96% coverage. New tests in tests/test_fallbacks.py cover:
  • ask_the_web when no web search is configured or SearXNG is down;
  • no ask_the_web when the fallback is off, not needed, or SearXNG answered;
  • leads parsing, de-duplication, held marking, and making no network call;
  • the CLI output and the MCP tool.
• ruff check, ruff format --check and pyright are clean.
• Live on this machine (no SearXNG): a no-hit search printed the ask-the-web prompt. leads on a DOI URL and a PubMed URL returned both identifiers marked as held, and returned a URL with no identifier under no_identifier.

🤖 Generated with Claude Code

Summary

A search already asks every provider and merges whatever answers, with web search last. This PR fixes how a failing provider is treated, by telling two kinds of failure apart.

An outside problem means the service is down, rate-limiting us or out of allowance.

• The provider is reported as unavailable and the other providers answer.
• The provider is then left alone for a while instead of being asked again by every search:
  • 60 s after an outage
  • 300 s after a 429
  • an hour after OpenAlex’s daily allowance
  • the rest of the process after scite’s monthly one
• While paused it isn’t called. Its report says unavailable with paused: true and the original reason.
• A cached answer is still served, and refresh=True asks anyway.

A defect means the service rejected our request or its answer changed shape. Waiting won’t fix it, so:

• It is never paused.
• Its report says error with broken: true.
• The search result carries a top-level broken list.
• The CLI prints ! BROKEN: … on stderr.
• The MCP server’s instructions tell the model to tell the user.
• The traceback is logged to a local file, shown by paper-fetch problems --trace.

The bug this fixes

Nine query helpers did if r.status != 200: return []. A provider answering 500, or rejecting a malformed query with 400, was reported as ok, 0 hits, so an outage or a bug read as “nothing in the literature”.

They now go through Provider._answered, which raises one kind of failure or the other. It applies only to query endpoints, which answer 200 with an empty list when nothing matches. A lookup by identifier can still treat a 404 as “not known”.

Also in this PR

• scite: its failures are sorted the same way. A spent allowance or a busy service is unavailable. A rejected request, a JSON-RPC error, an unrecognised tool error or an answer without hits is a defect.
• OpenAlex: a 429 or 5xx is unavailable rather than a generic error.
• Locating copies: a defect met while locating a copy is logged too.
• README install lines: they used a PyPI name that doesn’t exist (404). They now install from this repository.

Test plan

• uv run pytest: 367 passed, 96% coverage. Five existing tests relied on a 404 meaning “empty” and now use real empty answers.
  • tests/test_fallbacks.py covers pausing, the pause lengths, cache served while paused, defects named and logged, and web search asked when every provider is down.
  • tests/test_providers.py checks that all eight query providers treat a 503 or 403 as unavailable and a 400 as broken.
  • tests/test_scite.py covers the scite classification.
• ruff check, ruff format --check and pyright are clean.
• Live: five searches in one process. All nine default providers answered ok, including for a nonsense query, so none had been failing silently. scite reported its spent allowance once and was reported as paused on the next search. Zero defects were logged.

🤖 Generated with Claude Code

Summary

Makes scite opt-in. Its subscription allows 250 MCP calls a month, and one day in the default search set spent all of it.

After #7, every paper-fetch search asked scite, and a profile search asked it up to three times (once per query variant). With the research-rag gateway and about a dozen sessions searching, the month’s allowance was gone within a day. scite now answers “You have reached your monthly MCP usage limit (250 calls)”.

What changes

• Default set: scite is no longer in DEFAULT_SEARCH. An ordinary search doesn’t touch it.
• Opting in: a search asks scite on top of the usual providers with also=["scite"] in Python, --scite on the CLI, or scite: true on the MCP search tool. Naming it in PAPER_FETCH_SEARCH_PROVIDERS or providers= still works.
• One call per search: Provider.metered is new and is True for scite. A metered provider is asked the query as written, once, and never a profile’s variants.
• Session reuse: the opted-in provider is built once per Library, so scite’s MCP session is reused instead of opened per search. One thread uses that session at a time.
• Docs: README and CLAUDE.md record the limit and how to opt in.

Test plan

• uv run pytest: 329 passed, 95% coverage. New tests cover:
  • scite is asked only when told to;
  • also doesn’t ask a provider twice and reuses the session;
  • a metered provider is skipped for profile variants;
  • the CLI flag and the MCP parameter pass the opt-in through.
• ruff check, ruff format --check and pyright are clean.
• Live: a default paper-fetch search reports the nine usual providers and no scite.
• Live: --scite asks scite by name. It currently returns scite’s monthly-limit message, reported as unavailable.
• A successful live scite search through the opt-in path. This can’t run until the allowance resets.

🤖 Generated with Claude Code

Follow-up to #8. An MCP server runs every tool call on a worker thread, so one Library is used from several threads at once. #8 locked the passage index’s connection. This PR covers the rest of the state those threads share.

• Catalogue:
  • One lock guards the in-memory dictionary and its save.
  • Readers that walk the catalogue take a snapshot (_rows()). Writers go through _record / _forget.
  • A save now updates the dictionary in place without emptying it. Before, a lookup from another thread mid-save could find a held paper missing and fetch it again.
  • rebuild_index builds the new catalogue aside and swaps it in whole.
• Search memory and collections: each read-modify-write holds a lock, so no thread’s row or member is lost.
• Lazily built parts (profiles, memory, passage index, reranker): built once.
• Provider request spacing: two threads could both pass the wait at once and break a rate limit such as PLOS’s 10 per minute. The wait is now taken under a per-provider lock.

No lock is held across a fetch or a search, so slow network work does not block other calls. The locks are per process; the cross-process behaviour is unchanged.

Tested:

• tests/test_threads.py runs each part from 16 threads. All five tests fail on main and pass here.
• The thread tests passed 25 runs in a row.
• All 324 tests pass, and ruff and pyright are clean.

🤖 Generated with Claude Code

What failed: retrieve failed intermittently in service. The gateway log shows 13 failed calls. Eleven are sqlite3.InterfaceError: bad parameter or other API misuse, and one is passage text read back as undecodable bytes.

Cause: the MCP library runs every sync tool on a worker thread (anyio.to_thread.run_sync), so overlapping retrieve calls used the passage index’s single SQLite connection at the same time. The index was written as if calls were serial.

Fix: PassageIndex holds an RLock around every use of its connection, including reads, writes and the cached vector matrix. Embedding, the slow part of indexing, stays outside the lock, so indexing a paper does not block retrieval for seconds.

Tested:

• A new test runs 200 mixed search / passages / paper / stats / add calls across 16 threads on one index. It fails on main and passes here.
• All 319 tests pass, and ruff and pyright are clean.

Not covered: Library’s other shared state (the in-memory catalogue) is also reached from several threads. No failure from it appears in the logs, so this PR leaves it alone.

🤖 Generated with Claude Code

Why

A repo that pins the organisation’s private repos by git URL in pyproject.toml fails at uv sync in GitHub Actions (fatal: could not read Username for 'https://github.com': terminal prompts disabled), because the default GITHUB_TOKEN reads only its own repo. make check is then skipped, so the scaffold check never runs. This was worked around by hand in polarizetech/matched-entropy-demodulation#11.

What

• Templates (study, sim, tool, workbench): the workflow reads an optional PRIVATE_DEPS_TOKEN secret and, when it is set, rewrites https://github.com/{{org}}/ to use it. The org comes from the profile, not hard-coded.
• Install step: always runs uv sync --frozen || uv sync. A failure becomes a ::warning:: only when the token is unset and pyproject.toml or uv.lock mentions github.com/<org>/; any other failed install still fails the job.
• Docs: the secret is described in PROTOCOL.md § 4; UPGRADING.md and CHANGELOG.md have their v0.8.1 sections.
• Version: 0.8.1 in bin/scaffold, CITATION.cff and both plugin.json files.

Difference from the hand fix

matched-entropy-demodulation#11 skips uv sync whenever the token is unset. This does not, so a repo with only public dependencies and no secret keeps installing as before.

Upgrading from v0.8.0

• An unedited v0.8.0 workflow is replaced by scaffold update (by lock hash, or without a lock by matching what v0.8.0 wrote, which needs the scaffold’s git history and tag).
• An edited workflow keeps its edits and only the ref: moves; --adopt .github/workflows/check.yml replaces it.

Tests

python3 -m unittest discover tests: 67 run, 1 skipped (needs Python 3.11+), all pass locally. New tests cover the rendered org per profile, the install step’s three outcomes with a stand-in uv, and the v0.8.0 upgrade with and without a lock. One existing test changed: it asserted no {{ anywhere in a new repo, which ${{ secrets.… }} trips; it now looks for unfilled {{placeholder}} patterns.

The generated workflow has not yet run in GitHub Actions.

Not in this PR

Tagging v0.8.1 after merge. With private pins and no secret, make check still fails afterwards in a repo that has tests/, since uv run pytest needs the dependencies.

🤖 Generated with Claude Code

Summary

Adds scite as a search provider alongside OpenAlex and the others. It uses your scite subscription and needs no API key.

scite’s REST search (/api_partner/search) returns 401 without a partner token, which comes from their sales team. Its MCP server (https://api.scite.ai/mcp) accepts the subscriber’s own sign-in instead: OAuth 2.1 with dynamic client registration and PKCE. So the provider is a small MCP client written against Http. It sends initialize, then tools/call search_literature.

What changes

• Sign-in: paper-fetch scite-login opens the browser once and stores the sign-in in ~/.config/paper-fetch/scite-oauth.json (mode 0600). It is never written to the shared store. scite-logout removes it.
• Token renewal: the access token is renewed automatically. scite rotates the refresh token on every use, so the file is re-read before a refresh and once more after a refused one.
• Provider: providers/scite.py is search-only, like doaj. It is in the default search set and reported skipped until someone signs in. Its results are not cached.
• What a hit keeps: DOI, title, year, three authors, scite’s open-access flag and status, the citation tally and editorial notices.
• What a hit drops: scite’s access links (they carry the account’s email address), abstracts and citation statements.
• Merged search results: a hit that scite found carries a scite key with the tally, oa_status and editorial_notices.
• HTTP layer: Http.post is new. It returns a 4xx response as an answer, the way get does. http.py stays the only module that calls urlopen.
• Docs: README, CLAUDE.md and INTEGRATION.md are updated.

Design notes

• No mcp SDK in the core. The core stays standard-library plus pypdf, and all outbound requests still go through Http. The one new socket is the loopback listener that receives the browser redirect during login.
• oa_only: the tool has no open-access filter. The provider asks for three times limit (capped at 50, because every hit carries a citation list) and drops what scite marks closed.
• Failures: an expired sign-in, a 429, a tool error or an unreachable server is reported as unavailable with what to do, never as zero results.

Test plan

• uv run pytest: 318 passed, 95% coverage (29 new tests in tests/test_scite.py, against a fake scite server).
• ruff check, ruff format --check and pyright are clean.
• Live sign-in through the browser on 2026-10-05; the stored file is mode 0600.
• Live search: 5 hits in 3.7 s (3 POSTs on a new session), with nothing from the access links kept.
• Live oa_only search: every hit returned is open.
• Live token refresh: forced by expiring the token locally. The access token changed and the refresh token rotated.
• Live federated search: paper-fetch search … --providers scite,openalex,europepmc merged hits across the three providers.

🤖 Generated with Claude Code

dataset-fetch now has recommend(topic) (polarizetech/dataset-fetch#12). Choosing catalogues, phrasing queries and ranking datasets now live there, the way paper search lives in paper-fetch. This switches the datasets companion to that one call.

• Calls: one recommend call per planner query, at most two.
• Merging: cards are merged by reference, keeping the better score. Records with verified data files come first, then each by dataset-fetch’s own score.
• Printing: the answer shows the tool’s own fields: catalogue and kind, licence, files, subjects, modalities, creators, DOI, a pinned reference, and the tool’s reason for listing it. Records that hold only documents, and lower-ranked records, are counted rather than listed.
• Failures: a query that recommend refuses doesn’t discard the other query’s results. A catalogue that was silent on every query is named in Limits.
• Removed: the companion’s own loop over catalogues, the Zenodo query rewrite, the card reading and the document filter.

Tested:

• 238 tests pass, and ruff and pyright are clean.
• Live: one query, about 19 s.
  • dataset-fetch asked OpenNeuro, PhysioNet, Zenodo and OSF, and skipped the other ten with reasons.
  • Every listed record held data, including PhysioNet blood-pressure databases and a structured-exercise dataset.
  • The old loop listed none of six.

Needs a dataset-fetch installed with recommend.

🤖 Generated with Claude Code