What changed in the code.

Found by a live run. One verify call generated 16,000 tokens without ever closing its JSON answer, hit the 600 s wall-clock limit, and ended the whole run. It happened twice.

• Output cap: every structured call is capped by PIPELINE_MAX_OUTPUT_TOKENS (default 2048), so a runaway call stops in about a minute.
• Narrower error: ModelOutputError, a subclass of LLMError, marks one failed call: out of time or tokens, or not JSON. An unreachable server or a client that stopped answering still raises LLMError and ends the run.
• Failed checks: a verifier call that fails this way is recorded as CHECK_FAILED and never counts as support.
• Unverified claims: a claim that no verifier answered for gets VERIFIER_FAILED. It is removed from the answer and counted in Limits. A real verdict from another check still decides the claim.

Tested:

• 230 tests pass, including tests for each case. ruff and pyright are clean.
• The live re-run completed. The two runaway checks were capped and reported in Limits, and the answer was produced.

🤖 Generated with Claude Code

Phase 3 of splitting the work between the two repositories. paper-fetch now owns everything about finding and indexing papers (polarizetech/paper-fetch#2, #3, #4). This repository tells it the field and the concepts, then does everything after a passage is found: reading it, judging it, and checking the claims made from it.

Changes

• Search: search takes the domain’s discipline profile (paper-fetch’s profile with the same slug) and an optional collection (--collection, PIPELINE_COLLECTION, or research_start(collection=)). Searches are remembered, and the run log lists earlier searches on the same concept.
• Planning: the planner reads the profile’s indexed terms, measures and search guidance. If the library has no such profile, the run says so.
• Fetching: the library’s relevance() decides the fetch order, and the library indexes the fetched texts.
• Retrieval: passages come from the library’s retrieve(), with stable ids (<work>#p<ord>) and the paper’s metadata. Evidence.passage_id is now that string id.
• Removed: index.py, the embedding calls, the ONNX reranker, and the numpy dependency. The local-model reranker stays. Study-design notes move to notes.sqlite, keyed by the text hash.
• Domains: each keeps its evidence half (taxonomy, measures, critique rules, prompt rules). Anchors, sources and search guidance are now paper-fetch profiles, and DomainPolicy.profile names the profile to use. The domain template ships a profile.toml, and DOMAINS.md explains both halves.
• Gateway: the rag__ tools read and index through the paper library’s MCP session. Evidence ids keep the same format.

Tests

• 225 tests pass, and ruff and pyright are clean.
• The tests run against a real in-memory paper-fetch Library. It’s a test-only dependency, pinned to a commit that is now on paper-fetch’s main.
• The gateway tests run paper-fetch’s real MCP server as the upstream, with every proxy pointed at a closed port so nothing can reach the network.
• New tests check that the profile reaches the planner, that profile and collection reach search, fetch and retrieval, and that a missing profile is noted.

Live migration (already done locally)

• The old index (735 papers, 33,852 passages, bge-m3 vectors) was copied into paper-fetch’s index without re-embedding. Re-splitting six sampled texts reproduced their passages exactly, and the text hashes matched.
• The 272 cached study-design notes were copied into notes.sqlite.
• A round trip through the real gateway worked: hybrid retrieval, then a passing citation check.

🤖 Generated with Claude Code

Two small additions that retrieval-augmented-generation needs before it can drop its own index and embedding calls:

• Each passage’s paper now includes route and format, so an application’s run log can say where the quoted copy came from.
• New Library.relevance(targets, texts) and MCP tool relevance: scores each text (for example a hit’s title) against the nearest target (for example a sub-question), from 0 to 1. It uses the configured embedder, or concept overlap without one.

Tests: 259 pass offline, and ruff and pyright are clean.

🤖 Generated with Claude Code

Phase 2 of moving paper searching and indexing into paper-fetch. This PR is stacked on #2 (the base is search-profiles-collections), so merge #2 first.

What moves here from retrieval-augmented-generation

• The passage index (passages.py). It is a local SQLite file built from held full texts, with FTS5 (BM25) always and dense vectors when PAPER_FETCH_EMBED_MODEL is set, fused by reciprocal rank.
• Chunking and hidden-character cleaning are unchanged, so passage offsets and text_sha256 mean what they did in the RAG index. The paper row now also records how many hidden characters were removed (hidden_chars).
• Passage ids have the form <work>#p<ord> and survive rebuilds. sha identifies the passage text.
• The ONNX cross-encoder reranker (rerank.py), set with PAPER_FETCH_RERANK_MODEL.
• The embedder and the reranker are both deterministic: they return vectors and scores, never text. The model-based reranker and the prompt-injection scan stay in the application, because both concern a model reading the text.

API

• Library.retrieve(queries, identifiers=, collection=, limit, per_paper, rerank) indexes its scope on demand before retrieving.
• Also index_works, paper_passages, and the passages / reranker properties.
• New MCP tools: retrieve, passages, index. New CLI commands: paper-fetch index [--rebuild] and paper-fetch retrieve.
• New optional extras: retrieval (numpy) and rerank (onnxruntime, tokenizers, huggingface_hub). The core still depends only on pypdf and fonttools.
• Embedding requests go through the new Http.post_json, so the “only http.py and the store open URLs” guarantee still holds.

Tested:

• 257 tests pass offline: exact offsets, stable ids across rebuilds, hybrid and lexical ranking, scope, the per-paper cap, reranking, refusing a different embedding model, and the MCP and CLI commands. ruff and pyright are clean.
• A live run with bge-m3 against the real library, writing the index to a scratch file, turned 15 held papers into 1,405 passages in 94 s. A repeat query took about 10 ms.

Next (phase 3): retrieval-augmented-generation drops its own index and reranker and calls these tools, and its domains keep only their evidence rules.

🤖 Generated with Claude Code

Phase 1 of moving paper searching and indexing out of retrieval-augmented-generation and into paper-fetch. Nothing here uses a language model: everything is string matching and bookkeeping, so it is repeatable and tested offline.

Discipline profiles (profiles.py, data in profile_data/*.toml)

• A profile holds the search half of a field: controlled terms (MeSH ids are validated), synonyms, measures, sources, and search guidance for whoever writes the queries.
• search(profile=) also runs up to two variants that swap a synonym for its indexed term (“HRV” → “heart rate variability”). It never swaps the other way. Hits whose titles name the field’s terms rank first (profile_match), and results are merged across variants (found_by).
• Four profiles ship built in. They were converted from the RAG repo’s domains. They don’t narrow the provider set, because that would drop the repository providers that find open copies.
• A TOML file in ~/.config/paper-fetch/profiles/ adds a profile or replaces a built-in one by slug.

Named collections (collection.py)

• papers/collections/<name>.json holds a member list and a default profile. A paper is stored once, whichever collections list it.
• fetch(collection=) lists the paper in the collection. collect and uncollect add or remove members by any identifier.

Search memory (memory.py)

• papers/memory/searches.jsonl gets one row per search, written with the same re-read-and-append approach as the catalogue.
• Every search reports up to three earlier searches on the same concept, and marks each hit that an earlier search already found (seen_before).
• recall looks up past searches by concept, profile, collection or paper. A concept is the set of content words with synonyms mapped to indexed terms.

Surfaces

• New MCP tools: profiles, recall, collections, collection, create_collection, collect, uncollect. search and fetch take profile and collection.
• New CLI commands: profiles, collection, collect, recall, plus --profile and --collection on search and fetch.

Tested:

• 239 tests pass offline, including concurrent writers to the memory and to a collection. ruff and pyright are clean.
• A live search with the cardiovascular profile, run on an in-memory store, found the “heart rate variability” variant hits, ranked them first, and recalled the earlier “HRV” search at a score of 0.8.

Next:

• Phase 2: the passage index and retrieval move here.
• Phase 3: retrieval-augmented-generation calls paper-fetch for search, memory and retrieval, and its domains keep only the evidence half.

🤖 Generated with Claude Code

One gateway binary can now front a different set of upstreams under its own name. A second instance (the org-status MCP scientific-research-news) can then reuse the OAuth server instead of copying oauth.py.

• New optional server section in the gateway config: name, instructions, and builtin_tools. With builtin_tools: false the instance drops web__search and the rag__* tools, so it lists only gateway_status and its upstreams.
• Omitting the section keeps today’s behaviour exactly.
• Tests: one new test in tests/test_gateway.py; 9/9 pass. Ruff is clean. Documented in docs/mcp-clients.md.

🤖 Generated with Claude Code

Summary

Brings over the three docs from the audio-projects monorepo’s tools/paper-library (package paperlib), which is being replaced by this repo (polarizetech/audio-projects#26). dataset-fetch keeps the same three files at its root:

• CLAUDE.md: measured facts, gotchas and design decisions, with their dates.
• HARVEST.md: the procedure for bulk-collecting a topic into the library.
• INTEGRATION.md: how a consumer depends on paper-fetch. It covers the install from git with a pinned commit, setting up the S3/Spaces store through PAPER_FETCH_* in ~/.config/paper-fetch/*.env, the Python API and the MCP server.

Names, commands, env vars and APIs are rewritten to paper-fetch’s own and checked against src/paper_fetch/. Behaviour that has changed since the monorepo copy is described as it is now:

• Catalogue writes: since bee1bfe a save merges this process’s changes instead of the last writer winning.
• Cache: it lives under PAPER_FETCH_CACHE and applies only to the S3 store.
• load_env(): Library.default() does not call it, so Python consumers must.

Monorepo-only material is dropped: the tools/.venv-sci commands, sync.py and the generated copy, mcpkit envelopes, and REGISTRY/ORG-PROTOCOL references. Live measurements from the monorepo copy are kept, dated and labelled as such.

Docs only; no code changes.

Test plan

• Every command, flag and env var named in the docs exists in src/paper_fetch/.
• uv run pytest passes (214 tests).

🤖 Generated with Claude Code

A lab’s front door, in both renderers, drawn from the polarize.tech homepage redesign.

What’s added

• landing.css (zero-build) and src/components/landing.tsx (React), rendering the same classes: TopNav, Hero, Cta, Section, ProcessSteps, Promises, ProjectFeature, RepoCard, ChangeFeed, FeaturedArticle, ArticleCard, MetaLine, StoryCard / StoryRow / StoryGrid, Plate, Principle, TierLegend, SiteFooter. Responsive to phone width; the top bar folds into a <details> menu with no script.
• cellfield.js: <ui-cellfield>, a WebGL speck field (clusters at different depths, per-cell polarisation, occasional coordinated waves, an optional hero cell). Decoration only: aria-hidden, no fetch, one still frame under prefers-reduced-motion. React: <CellField>, same engine, typed by cellfield.d.ts.
• Token: radius.xs (4px).
• Checks: dev_check.py holds landing.css to the publication.css rules (generalised as check_layer) and checks cellfield.js; scripts/api_surface.py lists both files.
• Storybook: General UI/Landing: the full homepage at desktop and phone, plus each part.

Versioning

Additive only: nothing renamed or removed (api_surface.py --removed v0.5.5 is empty), so this is a patch release.

Checks

• python3 dev_check.py: 309/309
• build_css.py --check, build_shadcn.py --check: up to date
• tsc -p .: clean

polarize.tech adopts it in polarizetech/polarize.tech (branch home-redesign) once this is released.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 [email protected]

Single-cell habituation package: published Stentor models (Rajan & Marshall 2025 receptor inactivation; Rajan 2022 two-state; Eckert 2024 CIFF) in Tellurium/libRoadRunner (SBML), each reproduced unmodified first, plus minimal switchable changes (two input channels, membrane readout, synthesis block, Wood gating variant) and a trace exporter. Plan: see PR description updates. No experiment runs here.

🤖 Generated with Claude Code