paper-fetch

Summary

The last resort after the scholarly providers is a web search. Until now that needed a hosted SearXNG instance; without one, the fallback was skipped.

paper-fetch runs no language model, but its caller is usually a model with its own web search. This PR hands the last step to the caller, so nothing needs hosting.

• ask_the_web in search results. When the fallback is wanted (no open-access hit) and the library’s own web search did not answer, the result carries ask_the_web with why, the query, what happened to the library’s own web search, and how to proceed. “Did not answer” means SEARXNG_URL is unset or SearXNG is down.
• leads, available as Library.leads, the MCP leads tool and paper-fetch leads <url> .... It takes the URLs the caller found. It reads a DOI, PMID, PMCID or arXiv id out of each, using the web provider’s own parser, marks what is already held, and returns what to fetch. It makes no network calls. A URL with no identifier is returned under no_identifier rather than dropped.
• MCP instructions tell the model what to do when it sees ask_the_web.

The rule is unchanged: nothing is downloaded from a web page. An identifier goes through the open-access providers, which decide whether a copy may be kept.

SearXNG still works when configured, and then nothing is asked of the caller. web_fallback=False and PAPER_FETCH_WEB_FALLBACK=0 turn off the request along with the fallback.

Test plan

• uv run pytest: 375 passed, 96% coverage. New tests in tests/test_fallbacks.py cover:
  • ask_the_web when no web search is configured or SearXNG is down;
  • no ask_the_web when the fallback is off, not needed, or SearXNG answered;
  • leads parsing, de-duplication, held marking, and making no network call;
  • the CLI output and the MCP tool.
• ruff check, ruff format --check and pyright are clean.
• Live on this machine (no SearXNG): a no-hit search printed the ask-the-web prompt. leads on a DOI URL and a PubMed URL returned both identifiers marked as held, and returned a URL with no identifier under no_identifier.

🤖 Generated with Claude Code

Summary

A search already asks every provider and merges whatever answers, with web search last. This PR fixes how a failing provider is treated, by telling two kinds of failure apart.

An outside problem means the service is down, rate-limiting us or out of allowance.

• The provider is reported as unavailable and the other providers answer.
• The provider is then left alone for a while instead of being asked again by every search:
  • 60 s after an outage
  • 300 s after a 429
  • an hour after OpenAlex’s daily allowance
  • the rest of the process after scite’s monthly one
• While paused it isn’t called. Its report says unavailable with paused: true and the original reason.
• A cached answer is still served, and refresh=True asks anyway.

A defect means the service rejected our request or its answer changed shape. Waiting won’t fix it, so:

• It is never paused.
• Its report says error with broken: true.
• The search result carries a top-level broken list.
• The CLI prints ! BROKEN: … on stderr.
• The MCP server’s instructions tell the model to tell the user.
• The traceback is logged to a local file, shown by paper-fetch problems --trace.

The bug this fixes

Nine query helpers did if r.status != 200: return []. A provider answering 500, or rejecting a malformed query with 400, was reported as ok, 0 hits, so an outage or a bug read as “nothing in the literature”.

They now go through Provider._answered, which raises one kind of failure or the other. It applies only to query endpoints, which answer 200 with an empty list when nothing matches. A lookup by identifier can still treat a 404 as “not known”.

Also in this PR

• scite: its failures are sorted the same way. A spent allowance or a busy service is unavailable. A rejected request, a JSON-RPC error, an unrecognised tool error or an answer without hits is a defect.
• OpenAlex: a 429 or 5xx is unavailable rather than a generic error.
• Locating copies: a defect met while locating a copy is logged too.
• README install lines: they used a PyPI name that doesn’t exist (404). They now install from this repository.

Test plan

• uv run pytest: 367 passed, 96% coverage. Five existing tests relied on a 404 meaning “empty” and now use real empty answers.
  • tests/test_fallbacks.py covers pausing, the pause lengths, cache served while paused, defects named and logged, and web search asked when every provider is down.
  • tests/test_providers.py checks that all eight query providers treat a 503 or 403 as unavailable and a 400 as broken.
  • tests/test_scite.py covers the scite classification.
• ruff check, ruff format --check and pyright are clean.
• Live: five searches in one process. All nine default providers answered ok, including for a nonsense query, so none had been failing silently. scite reported its spent allowance once and was reported as paused on the next search. Zero defects were logged.

🤖 Generated with Claude Code

Summary

Makes scite opt-in. Its subscription allows 250 MCP calls a month, and one day in the default search set spent all of it.

After #7, every paper-fetch search asked scite, and a profile search asked it up to three times (once per query variant). With the research-rag gateway and about a dozen sessions searching, the month’s allowance was gone within a day. scite now answers “You have reached your monthly MCP usage limit (250 calls)”.

What changes

• Default set: scite is no longer in DEFAULT_SEARCH. An ordinary search doesn’t touch it.
• Opting in: a search asks scite on top of the usual providers with also=["scite"] in Python, --scite on the CLI, or scite: true on the MCP search tool. Naming it in PAPER_FETCH_SEARCH_PROVIDERS or providers= still works.
• One call per search: Provider.metered is new and is True for scite. A metered provider is asked the query as written, once, and never a profile’s variants.
• Session reuse: the opted-in provider is built once per Library, so scite’s MCP session is reused instead of opened per search. One thread uses that session at a time.
• Docs: README and CLAUDE.md record the limit and how to opt in.

Test plan

• uv run pytest: 329 passed, 95% coverage. New tests cover:
  • scite is asked only when told to;
  • also doesn’t ask a provider twice and reuses the session;
  • a metered provider is skipped for profile variants;
  • the CLI flag and the MCP parameter pass the opt-in through.
• ruff check, ruff format --check and pyright are clean.
• Live: a default paper-fetch search reports the nine usual providers and no scite.
• Live: --scite asks scite by name. It currently returns scite’s monthly-limit message, reported as unavailable.
• A successful live scite search through the opt-in path. This can’t run until the allowance resets.

🤖 Generated with Claude Code

Follow-up to #8. An MCP server runs every tool call on a worker thread, so one Library is used from several threads at once. #8 locked the passage index’s connection. This PR covers the rest of the state those threads share.

• Catalogue:
  • One lock guards the in-memory dictionary and its save.
  • Readers that walk the catalogue take a snapshot (_rows()). Writers go through _record / _forget.
  • A save now updates the dictionary in place without emptying it. Before, a lookup from another thread mid-save could find a held paper missing and fetch it again.
  • rebuild_index builds the new catalogue aside and swaps it in whole.
• Search memory and collections: each read-modify-write holds a lock, so no thread’s row or member is lost.
• Lazily built parts (profiles, memory, passage index, reranker): built once.
• Provider request spacing: two threads could both pass the wait at once and break a rate limit such as PLOS’s 10 per minute. The wait is now taken under a per-provider lock.

No lock is held across a fetch or a search, so slow network work does not block other calls. The locks are per process; the cross-process behaviour is unchanged.

Tested:

• tests/test_threads.py runs each part from 16 threads. All five tests fail on main and pass here.
• The thread tests passed 25 runs in a row.
• All 324 tests pass, and ruff and pyright are clean.

🤖 Generated with Claude Code

What failed: retrieve failed intermittently in service. The gateway log shows 13 failed calls. Eleven are sqlite3.InterfaceError: bad parameter or other API misuse, and one is passage text read back as undecodable bytes.

Cause: the MCP library runs every sync tool on a worker thread (anyio.to_thread.run_sync), so overlapping retrieve calls used the passage index’s single SQLite connection at the same time. The index was written as if calls were serial.

Fix: PassageIndex holds an RLock around every use of its connection, including reads, writes and the cached vector matrix. Embedding, the slow part of indexing, stays outside the lock, so indexing a paper does not block retrieval for seconds.

Tested:

• A new test runs 200 mixed search / passages / paper / stats / add calls across 16 threads on one index. It fails on main and passes here.
• All 319 tests pass, and ruff and pyright are clean.

Not covered: Library’s other shared state (the in-memory catalogue) is also reached from several threads. No failure from it appears in the logs, so this PR leaves it alone.

🤖 Generated with Claude Code

Summary

Adds scite as a search provider alongside OpenAlex and the others. It uses your scite subscription and needs no API key.

scite’s REST search (/api_partner/search) returns 401 without a partner token, which comes from their sales team. Its MCP server (https://api.scite.ai/mcp) accepts the subscriber’s own sign-in instead: OAuth 2.1 with dynamic client registration and PKCE. So the provider is a small MCP client written against Http. It sends initialize, then tools/call search_literature.

What changes

• Sign-in: paper-fetch scite-login opens the browser once and stores the sign-in in ~/.config/paper-fetch/scite-oauth.json (mode 0600). It is never written to the shared store. scite-logout removes it.
• Token renewal: the access token is renewed automatically. scite rotates the refresh token on every use, so the file is re-read before a refresh and once more after a refused one.
• Provider: providers/scite.py is search-only, like doaj. It is in the default search set and reported skipped until someone signs in. Its results are not cached.
• What a hit keeps: DOI, title, year, three authors, scite’s open-access flag and status, the citation tally and editorial notices.
• What a hit drops: scite’s access links (they carry the account’s email address), abstracts and citation statements.
• Merged search results: a hit that scite found carries a scite key with the tally, oa_status and editorial_notices.
• HTTP layer: Http.post is new. It returns a 4xx response as an answer, the way get does. http.py stays the only module that calls urlopen.
• Docs: README, CLAUDE.md and INTEGRATION.md are updated.

Design notes

• No mcp SDK in the core. The core stays standard-library plus pypdf, and all outbound requests still go through Http. The one new socket is the loopback listener that receives the browser redirect during login.
• oa_only: the tool has no open-access filter. The provider asks for three times limit (capped at 50, because every hit carries a citation list) and drops what scite marks closed.
• Failures: an expired sign-in, a 429, a tool error or an unreachable server is reported as unavailable with what to do, never as zero results.

Test plan

• uv run pytest: 318 passed, 95% coverage (29 new tests in tests/test_scite.py, against a fake scite server).
• ruff check, ruff format --check and pyright are clean.
• Live sign-in through the browser on 2026-10-05; the stored file is mode 0600.
• Live search: 5 hits in 3.7 s (3 POSTs on a new session), with nothing from the access links kept.
• Live oa_only search: every hit returned is open.
• Live token refresh: forced by expiring the token locally. The access token changed and the refresh token rotated.
• Live federated search: paper-fetch search … --providers scite,openalex,europepmc merged hits across the three providers.

🤖 Generated with Claude Code

Found in a live run of retrieval-augmented-generation: a bibliography entry from a PDF was retrieved as a passage. The chunker only cut at a paragraph that was exactly “References”. PDF text often has the heading on a line inside a paragraph, or loses it altogether.

Detection (references_start)

• A heading on a line of its own (“References”, “6. References”, “Literature Cited”), past the first third of the text.
• Or, with no heading: a run of at least 8 consecutively numbered entries starting at 1 that begins in the second half of the text, reaches its last tenth, and mostly carries publication years.
• Numbered lists in the body are kept. That covers lists that are short, early, have no years, or do not start at 1.

Measured on 80 held papers: 11 of 12 PDFs are now cut (none were before), every cut I read lands at the start of the list, and the 68 JATS and TEI texts are unaffected. An unnumbered author-year list with no heading is still not recognised.

Existing indexes

• The chunker has a version (CHUNKER = 2), kept in a new chunked table next to the indexed_at of the row it describes.
• The papers table is unchanged, so servers still running older code and sharing the file keep working. A paper one of them re-indexes counts as stale again.
• has() is false for a stale paper, so it is re-chunked the next time it is indexed. Passages whose text did not change keep their vectors, so only new text is embedded.
• status reports stale.

Tested: 287 tests pass, including the detector’s positive and negative cases, vector reuse, an index from before versions existed, and a row rewritten by older code. ruff and pyright are clean.

🤖 Generated with Claude Code

Problems

1. Works the library creates itself (`pmcid-PMC5519140`, `doi-10.1214%2F12-aoas565supp`) were rejected by `index`, `collect` and the MCP `index`/`collect` tools with “unrecognised identifier”. One bad id also failed a whole batch.
2. `paper-fetch index` with no arguments aborted on the first unparseable record, leaving later works unindexed.

Changes

• `ids.normalize` parses `doi-`/`pmcid-`/`pmid-`/`arxiv-` work ids, including percent-encoded DOIs. Behaviour for ordinary DOI / OpenAlex / PMID / PMCID inputs is unchanged and junk is still refused.
• `index_works` reads text from the catalogue row (no re-parse) and skips and reports unreadable records under `skipped`; the CLI prints each with its reason and a count.
• `collect` returns a `rejected` list instead of failing the batch.
• When OpenAlex has no record, `fetch` fills title, year, authors and DOI from Europe PMC (never fatal).
• Existing library records are not rewritten; already-stored metadata-less works are only enriched on a new `fetch`.

Tests

`tests/test_own_work_ids.py`: 12 of its 19 cases fail on main and all pass here; full suite 278 passed, ruff and pyright clean.

🤖 Generated with Claude Code

Two small additions that retrieval-augmented-generation needs before it can drop its own index and embedding calls:

• Each passage’s paper now includes route and format, so an application’s run log can say where the quoted copy came from.
• New Library.relevance(targets, texts) and MCP tool relevance: scores each text (for example a hit’s title) against the nearest target (for example a sub-question), from 0 to 1. It uses the configured embedder, or concept overlap without one.

Tests: 259 pass offline, and ruff and pyright are clean.

🤖 Generated with Claude Code

Phase 2 of moving paper searching and indexing into paper-fetch. This PR is stacked on #2 (the base is search-profiles-collections), so merge #2 first.

What moves here from retrieval-augmented-generation

• The passage index (passages.py). It is a local SQLite file built from held full texts, with FTS5 (BM25) always and dense vectors when PAPER_FETCH_EMBED_MODEL is set, fused by reciprocal rank.
• Chunking and hidden-character cleaning are unchanged, so passage offsets and text_sha256 mean what they did in the RAG index. The paper row now also records how many hidden characters were removed (hidden_chars).
• Passage ids have the form <work>#p<ord> and survive rebuilds. sha identifies the passage text.
• The ONNX cross-encoder reranker (rerank.py), set with PAPER_FETCH_RERANK_MODEL.
• The embedder and the reranker are both deterministic: they return vectors and scores, never text. The model-based reranker and the prompt-injection scan stay in the application, because both concern a model reading the text.

API

• Library.retrieve(queries, identifiers=, collection=, limit, per_paper, rerank) indexes its scope on demand before retrieving.
• Also index_works, paper_passages, and the passages / reranker properties.
• New MCP tools: retrieve, passages, index. New CLI commands: paper-fetch index [--rebuild] and paper-fetch retrieve.
• New optional extras: retrieval (numpy) and rerank (onnxruntime, tokenizers, huggingface_hub). The core still depends only on pypdf and fonttools.
• Embedding requests go through the new Http.post_json, so the “only http.py and the store open URLs” guarantee still holds.

Tested:

• 257 tests pass offline: exact offsets, stable ids across rebuilds, hybrid and lexical ranking, scope, the per-paper cap, reranking, refusing a different embedding model, and the MCP and CLI commands. ruff and pyright are clean.
• A live run with bge-m3 against the real library, writing the index to a scratch file, turned 15 held papers into 1,405 passages in 94 s. A repeat query took about 10 ms.

Next (phase 3): retrieval-augmented-generation drops its own index and reranker and calls these tools, and its domains keep only their evidence rules.

🤖 Generated with Claude Code

Phase 1 of moving paper searching and indexing out of retrieval-augmented-generation and into paper-fetch. Nothing here uses a language model: everything is string matching and bookkeeping, so it is repeatable and tested offline.

Discipline profiles (profiles.py, data in profile_data/*.toml)

• A profile holds the search half of a field: controlled terms (MeSH ids are validated), synonyms, measures, sources, and search guidance for whoever writes the queries.
• search(profile=) also runs up to two variants that swap a synonym for its indexed term (“HRV” → “heart rate variability”). It never swaps the other way. Hits whose titles name the field’s terms rank first (profile_match), and results are merged across variants (found_by).
• Four profiles ship built in. They were converted from the RAG repo’s domains. They don’t narrow the provider set, because that would drop the repository providers that find open copies.
• A TOML file in ~/.config/paper-fetch/profiles/ adds a profile or replaces a built-in one by slug.

Named collections (collection.py)

• papers/collections/<name>.json holds a member list and a default profile. A paper is stored once, whichever collections list it.
• fetch(collection=) lists the paper in the collection. collect and uncollect add or remove members by any identifier.

Search memory (memory.py)

• papers/memory/searches.jsonl gets one row per search, written with the same re-read-and-append approach as the catalogue.
• Every search reports up to three earlier searches on the same concept, and marks each hit that an earlier search already found (seen_before).
• recall looks up past searches by concept, profile, collection or paper. A concept is the set of content words with synonyms mapped to indexed terms.

Surfaces

• New MCP tools: profiles, recall, collections, collection, create_collection, collect, uncollect. search and fetch take profile and collection.
• New CLI commands: profiles, collection, collect, recall, plus --profile and --collection on search and fetch.

Tested:

• 239 tests pass offline, including concurrent writers to the memory and to a collection. ruff and pyright are clean.
• A live search with the cardiovascular profile, run on an in-memory store, found the “heart rate variability” variant hits, ranked them first, and recalled the earlier “HRV” search at a score of 0.8.

Next:

• Phase 2: the passage index and retrieval move here.
• Phase 3: retrieval-augmented-generation calls paper-fetch for search, memory and retrieval, and its domains keep only the evidence half.

🤖 Generated with Claude Code

Summary

Brings over the three docs from the audio-projects monorepo’s tools/paper-library (package paperlib), which is being replaced by this repo (polarizetech/audio-projects#26). dataset-fetch keeps the same three files at its root:

• CLAUDE.md: measured facts, gotchas and design decisions, with their dates.
• HARVEST.md: the procedure for bulk-collecting a topic into the library.
• INTEGRATION.md: how a consumer depends on paper-fetch. It covers the install from git with a pinned commit, setting up the S3/Spaces store through PAPER_FETCH_* in ~/.config/paper-fetch/*.env, the Python API and the MCP server.

Names, commands, env vars and APIs are rewritten to paper-fetch’s own and checked against src/paper_fetch/. Behaviour that has changed since the monorepo copy is described as it is now:

• Catalogue writes: since bee1bfe a save merges this process’s changes instead of the last writer winning.
• Cache: it lives under PAPER_FETCH_CACHE and applies only to the S3 store.
• load_env(): Library.default() does not call it, so Python consumers must.

Monorepo-only material is dropped: the tools/.venv-sci commands, sync.py and the generated copy, mcpkit envelopes, and REGISTRY/ORG-PROTOCOL references. Live measurements from the monorepo copy are kept, dated and labelled as such.

Docs only; no code changes.

Test plan

• Every command, flag and env var named in the docs exists in src/paper_fetch/.
• uv run pytest passes (214 tests).

🤖 Generated with Claude Code