What changed in the code.

Four changes that came out of live runs, plus the first companion tool.

One model call with no answer costs that item, not the run (follow-up to #3)

• Reading a passage: it is dropped with the flag no_answer_from_model and counted in Limits.
• Classifying a paper: it is graded unclear for this run, and that is not cached.
• Planning, synthesis and critique still end the run, as does an unreachable model.

Reranking is the paper library’s

• The local-model reranker is removed. In a live run the 4B model spent 4.9 of 5.7 hours reranking.
• The library’s cross-encoder (PAPER_FETCH_RERANK_MODEL) ranks passages. The pipeline keeps that order and only caps passages per paper and per sub-question.
• Without the setting, passages are read in retrieval order, and the run notes say so.

Two bugs from that run

• Fetch ordering now puts papers reported as open access first and papers reported as closed last. All six fetches had gone to closed papers.
• An exact quote match that stops inside a word or number is widened to the whole token. A stored quote read “5.” where the passage said “5.9 mmHg”.

Companion tools (research_pipeline/companions.py)

• With a companion configured under companions in the gateway config, the plan’s schema gains a tools field restricted to the configured keys.
• Code calls each requested tool once, within set bounds, and prints the tool’s own records in the answer. No model writes that section, and it never becomes evidence.
• A companion that fails is reported and the run goes on. Offline runs call none.
• First companion: datasets, which speaks dataset-fetch’s MCP contract.
  • It searches every catalogue that takes a free-text query, with Zenodo restricted to dataset-typed records.
  • It reads the data cards, and counts documents-only records instead of listing them.
  • It orders the rest by the library’s relevance.
• Adding another companion is one registry entry.

Also

• A generic Upstream MCP client; PaperLibrary is now one use of it.
• The design doc notes the planned “own endpoint” backend for private runs.

Tested:

• 237 tests pass. ruff and pyright are clean.
• Live against the real services: the library reranker returned on-topic passages, and the dataset companion searched three catalogues and read ten cards in 14 s.

🤖 Generated with Claude Code

Preregistration in progress. Stops at the -prereg tag for the operator to hear predictions before any run.

🤖 Generated with Claude Code

Found by a live run. One verify call generated 16,000 tokens without ever closing its JSON answer, hit the 600 s wall-clock limit, and ended the whole run. It happened twice.

• Output cap: every structured call is capped by PIPELINE_MAX_OUTPUT_TOKENS (default 2048), so a runaway call stops in about a minute.
• Narrower error: ModelOutputError, a subclass of LLMError, marks one failed call: out of time or tokens, or not JSON. An unreachable server or a client that stopped answering still raises LLMError and ends the run.
• Failed checks: a verifier call that fails this way is recorded as CHECK_FAILED and never counts as support.
• Unverified claims: a claim that no verifier answered for gets VERIFIER_FAILED. It is removed from the answer and counted in Limits. A real verdict from another check still decides the claim.

Tested:

• 230 tests pass, including tests for each case. ruff and pyright are clean.
• The live re-run completed. The two runaway checks were capped and reported in Limits, and the answer was produced.

🤖 Generated with Claude Code

Phase 3 of splitting the work between the two repositories. paper-fetch now owns everything about finding and indexing papers (polarizetech/paper-fetch#2, #3, #4). This repository tells it the field and the concepts, then does everything after a passage is found: reading it, judging it, and checking the claims made from it.

Changes

• Search: search takes the domain’s discipline profile (paper-fetch’s profile with the same slug) and an optional collection (--collection, PIPELINE_COLLECTION, or research_start(collection=)). Searches are remembered, and the run log lists earlier searches on the same concept.
• Planning: the planner reads the profile’s indexed terms, measures and search guidance. If the library has no such profile, the run says so.
• Fetching: the library’s relevance() decides the fetch order, and the library indexes the fetched texts.
• Retrieval: passages come from the library’s retrieve(), with stable ids (<work>#p<ord>) and the paper’s metadata. Evidence.passage_id is now that string id.
• Removed: index.py, the embedding calls, the ONNX reranker, and the numpy dependency. The local-model reranker stays. Study-design notes move to notes.sqlite, keyed by the text hash.
• Domains: each keeps its evidence half (taxonomy, measures, critique rules, prompt rules). Anchors, sources and search guidance are now paper-fetch profiles, and DomainPolicy.profile names the profile to use. The domain template ships a profile.toml, and DOMAINS.md explains both halves.
• Gateway: the rag__ tools read and index through the paper library’s MCP session. Evidence ids keep the same format.

Tests

• 225 tests pass, and ruff and pyright are clean.
• The tests run against a real in-memory paper-fetch Library. It’s a test-only dependency, pinned to a commit that is now on paper-fetch’s main.
• The gateway tests run paper-fetch’s real MCP server as the upstream, with every proxy pointed at a closed port so nothing can reach the network.
• New tests check that the profile reaches the planner, that profile and collection reach search, fetch and retrieval, and that a missing profile is noted.

Live migration (already done locally)

• The old index (735 papers, 33,852 passages, bge-m3 vectors) was copied into paper-fetch’s index without re-embedding. Re-splitting six sampled texts reproduced their passages exactly, and the text hashes matched.
• The 272 cached study-design notes were copied into notes.sqlite.
• A round trip through the real gateway worked: hybrid retrieval, then a passing citation check.

🤖 Generated with Claude Code

Two small additions that retrieval-augmented-generation needs before it can drop its own index and embedding calls:

• Each passage’s paper now includes route and format, so an application’s run log can say where the quoted copy came from.
• New Library.relevance(targets, texts) and MCP tool relevance: scores each text (for example a hit’s title) against the nearest target (for example a sub-question), from 0 to 1. It uses the configured embedder, or concept overlap without one.

Tests: 259 pass offline, and ruff and pyright are clean.

🤖 Generated with Claude Code

Phase 2 of moving paper searching and indexing into paper-fetch. This PR is stacked on #2 (the base is search-profiles-collections), so merge #2 first.

What moves here from retrieval-augmented-generation

• The passage index (passages.py). It is a local SQLite file built from held full texts, with FTS5 (BM25) always and dense vectors when PAPER_FETCH_EMBED_MODEL is set, fused by reciprocal rank.
• Chunking and hidden-character cleaning are unchanged, so passage offsets and text_sha256 mean what they did in the RAG index. The paper row now also records how many hidden characters were removed (hidden_chars).
• Passage ids have the form <work>#p<ord> and survive rebuilds. sha identifies the passage text.
• The ONNX cross-encoder reranker (rerank.py), set with PAPER_FETCH_RERANK_MODEL.
• The embedder and the reranker are both deterministic: they return vectors and scores, never text. The model-based reranker and the prompt-injection scan stay in the application, because both concern a model reading the text.

API

• Library.retrieve(queries, identifiers=, collection=, limit, per_paper, rerank) indexes its scope on demand before retrieving.
• Also index_works, paper_passages, and the passages / reranker properties.
• New MCP tools: retrieve, passages, index. New CLI commands: paper-fetch index [--rebuild] and paper-fetch retrieve.
• New optional extras: retrieval (numpy) and rerank (onnxruntime, tokenizers, huggingface_hub). The core still depends only on pypdf and fonttools.
• Embedding requests go through the new Http.post_json, so the “only http.py and the store open URLs” guarantee still holds.

Tested:

• 257 tests pass offline: exact offsets, stable ids across rebuilds, hybrid and lexical ranking, scope, the per-paper cap, reranking, refusing a different embedding model, and the MCP and CLI commands. ruff and pyright are clean.
• A live run with bge-m3 against the real library, writing the index to a scratch file, turned 15 held papers into 1,405 passages in 94 s. A repeat query took about 10 ms.

Next (phase 3): retrieval-augmented-generation drops its own index and reranker and calls these tools, and its domains keep only their evidence rules.

🤖 Generated with Claude Code

Phase 1 of moving paper searching and indexing out of retrieval-augmented-generation and into paper-fetch. Nothing here uses a language model: everything is string matching and bookkeeping, so it is repeatable and tested offline.

Discipline profiles (profiles.py, data in profile_data/*.toml)

• A profile holds the search half of a field: controlled terms (MeSH ids are validated), synonyms, measures, sources, and search guidance for whoever writes the queries.
• search(profile=) also runs up to two variants that swap a synonym for its indexed term (“HRV” → “heart rate variability”). It never swaps the other way. Hits whose titles name the field’s terms rank first (profile_match), and results are merged across variants (found_by).
• Four profiles ship built in. They were converted from the RAG repo’s domains. They don’t narrow the provider set, because that would drop the repository providers that find open copies.
• A TOML file in ~/.config/paper-fetch/profiles/ adds a profile or replaces a built-in one by slug.

Named collections (collection.py)

• papers/collections/<name>.json holds a member list and a default profile. A paper is stored once, whichever collections list it.
• fetch(collection=) lists the paper in the collection. collect and uncollect add or remove members by any identifier.

Search memory (memory.py)

• papers/memory/searches.jsonl gets one row per search, written with the same re-read-and-append approach as the catalogue.
• Every search reports up to three earlier searches on the same concept, and marks each hit that an earlier search already found (seen_before).
• recall looks up past searches by concept, profile, collection or paper. A concept is the set of content words with synonyms mapped to indexed terms.

Surfaces

• New MCP tools: profiles, recall, collections, collection, create_collection, collect, uncollect. search and fetch take profile and collection.
• New CLI commands: profiles, collection, collect, recall, plus --profile and --collection on search and fetch.

Tested:

• 239 tests pass offline, including concurrent writers to the memory and to a collection. ruff and pyright are clean.
• A live search with the cardiovascular profile, run on an in-memory store, found the “heart rate variability” variant hits, ranked them first, and recalled the earlier “HRV” search at a score of 0.8.

Next:

• Phase 2: the passage index and retrieval move here.
• Phase 3: retrieval-augmented-generation calls paper-fetch for search, memory and retrieval, and its domains keep only the evidence half.

🤖 Generated with Claude Code

One gateway binary can now front a different set of upstreams under its own name. A second instance (the org-status MCP scientific-research-news) can then reuse the OAuth server instead of copying oauth.py.

• New optional server section in the gateway config: name, instructions, and builtin_tools. With builtin_tools: false the instance drops web__search and the rag__* tools, so it lists only gateway_status and its upstreams.
• Omitting the section keeps today’s behaviour exactly.
• Tests: one new test in tests/test_gateway.py; 9/9 pass. Ruff is clean. Documented in docs/mcp-clients.md.

🤖 Generated with Claude Code

Summary

Brings over the three docs from the audio-projects monorepo’s tools/paper-library (package paperlib), which is being replaced by this repo (polarizetech/audio-projects#26). dataset-fetch keeps the same three files at its root:

• CLAUDE.md: measured facts, gotchas and design decisions, with their dates.
• HARVEST.md: the procedure for bulk-collecting a topic into the library.
• INTEGRATION.md: how a consumer depends on paper-fetch. It covers the install from git with a pinned commit, setting up the S3/Spaces store through PAPER_FETCH_* in ~/.config/paper-fetch/*.env, the Python API and the MCP server.

Names, commands, env vars and APIs are rewritten to paper-fetch’s own and checked against src/paper_fetch/. Behaviour that has changed since the monorepo copy is described as it is now:

• Catalogue writes: since bee1bfe a save merges this process’s changes instead of the last writer winning.
• Cache: it lives under PAPER_FETCH_CACHE and applies only to the S3 store.
• load_env(): Library.default() does not call it, so Python consumers must.

Monorepo-only material is dropped: the tools/.venv-sci commands, sync.py and the generated copy, mcpkit envelopes, and REGISTRY/ORG-PROTOCOL references. Live measurements from the monorepo copy are kept, dated and labelled as such.

Docs only; no code changes.

Test plan

• Every command, flag and env var named in the docs exists in src/paper_fetch/.
• uv run pytest passes (214 tests).

🤖 Generated with Claude Code

A lab’s front door, in both renderers, drawn from the polarize.tech homepage redesign.

What’s added

• landing.css (zero-build) and src/components/landing.tsx (React), rendering the same classes: TopNav, Hero, Cta, Section, ProcessSteps, Promises, ProjectFeature, RepoCard, ChangeFeed, FeaturedArticle, ArticleCard, MetaLine, StoryCard / StoryRow / StoryGrid, Plate, Principle, TierLegend, SiteFooter. Responsive to phone width; the top bar folds into a <details> menu with no script.
• cellfield.js: <ui-cellfield>, a WebGL speck field (clusters at different depths, per-cell polarisation, occasional coordinated waves, an optional hero cell). Decoration only: aria-hidden, no fetch, one still frame under prefers-reduced-motion. React: <CellField>, same engine, typed by cellfield.d.ts.
• Token: radius.xs (4px).
• Checks: dev_check.py holds landing.css to the publication.css rules (generalised as check_layer) and checks cellfield.js; scripts/api_surface.py lists both files.
• Storybook: General UI/Landing: the full homepage at desktop and phone, plus each part.

Versioning

Additive only: nothing renamed or removed (api_surface.py --removed v0.5.5 is empty), so this is a patch release.

Checks

• python3 dev_check.py: 309/309
• build_css.py --check, build_shadcn.py --check: up to date
• tsc -p .: clean

polarize.tech adopts it in polarizetech/polarize.tech (branch home-redesign) once this is released.

🤖 Generated with Claude Code