What changed in the code.

The bug

quality.spectrum() promised to summarise Goertzel bins into display buckets, but evaluated only every step-th DFT bin and skipped the rest. On a record longer than a couple of seconds a narrow line fell between the evaluated bins and disappeared. A 260-count 10 Hz sine in noise over 20 s drew its tallest point at 35.65 Hz, about 1.7 counts. The Monitor tab’s spectrum (POST /spectrum, up to 20 s every ~500 ms) was affected the same way.

The fix

• Every transform bin in the range is evaluated, and each bucket reports the maximum amplitude of the bins it covers. A maximum keeps a narrow line at its own amplitude however long the record is; a mean of power would dilute it. The price, stated in the docstring: the noise floor of the display rises slowly with record length, and it is not a power density.
• Cost: one stdlib radix-2 FFT of the Hann-windowed record, zero-padded to the next power of two at or above twice its length, over at most the most recent 16,384 samples (65 s at 250 Hz). The padding bounds the scalloping loss at 4 %.
• freqs are now bucket centres. resolution_hz keeps its meaning (rate / samples used). New fields: bucket_hz, bin_hz, samples_used, summary. The web client needs no change.

Checks

• apps/server/dev_check.py: a 10 Hz sine in noise at 2 s, 20 s and 60 s must be the tallest point, within one bucket of 10 Hz and within 10 % of the input amplitude; plus a check on the sample cap.
• python3 scripts/check.py: all seven suites pass. check_docs.py run against the blog post: in step. ruff clean.

Versioning

No version bump and no model-v* tag. quality.py is in the measuring chain, but spectrum() feeds only the drawing: nothing it returns is stored in a session or an export, and assess() and bands() are unchanged. The CHANGELOG entry says so. If this should be 0.2.1 instead, that is a follow-up commit here.

🤖 Generated with Claude Code

What this adds

A generated docs site for this repository, built by automatic-documentation-site. Nothing on it is written by hand: a page is fixed by fixing the Markdown file, docstring or story it names.

No code under apps/ changes.

What the site contains (22 pages)

• Guides: the README (with Safety and the board photos), then the hardware, recording, standards, versioning, bridge, workbench and project documents.
• API reference: apps/server and apps/web Python, and the exported browser modules. 85 public symbols, read without importing or running anything.
• Stories, on synthetic or modelled inputs only, never a recording of a person:
  • what the front end can carry (rig.carries): alpha reachable, brainstem responses unreachable
  • which readouts fit in its band (readouts.gate)
  • the pre-flight verdict on a good, a flat and a railed channel (quality.assess)
  • the delivered sample rate (timebase.RateEstimator recovers a simulated 248.3 Hz)

One story records a check that does not hold

quality.spectrum of a clean 10 Hz sine has no peak at 10 Hz on a 20 s record. Its docstring says it averages Goertzel bins into display buckets; the code evaluates every step-th bin, so a narrow line between them is missed. The story shows the check as it came out (“absent”) and was not adjusted to pass. The fix belongs in a separate change.

Checked

• docsite build . --refresh --strict --publish pages with the released tool (v0.2.1) and polarize-ui v0.5.15: 22 pages, 4/4 stories run.
• python3 scripts/check.py: every suite passes.
• The built site opens from a folder (file://) with plots, search and badges working.

After merging

The workflow runs on main and deploys to https://polarizetech.github.io/olimex-shield/ (GitHub Pages is enabled, source: GitHub Actions).

Based on main. The extension catalogue on the tool-only work is not on main yet; when it lands it is one [catalogue] block in docs-site.toml.

🤖 Generated with Claude Code

Found in a live run of retrieval-augmented-generation: a bibliography entry from a PDF was retrieved as a passage. The chunker only cut at a paragraph that was exactly “References”. PDF text often has the heading on a line inside a paragraph, or loses it altogether.

Detection (references_start)

• A heading on a line of its own (“References”, “6. References”, “Literature Cited”), past the first third of the text.
• Or, with no heading: a run of at least 8 consecutively numbered entries starting at 1 that begins in the second half of the text, reaches its last tenth, and mostly carries publication years.
• Numbered lists in the body are kept. That covers lists that are short, early, have no years, or do not start at 1.

Measured on 80 held papers: 11 of 12 PDFs are now cut (none were before), every cut I read lands at the start of the list, and the 68 JATS and TEI texts are unaffected. An unnumbered author-year list with no heading is still not recognised.

Existing indexes

• The chunker has a version (CHUNKER = 2), kept in a new chunked table next to the indexed_at of the row it describes.
• The papers table is unchanged, so servers still running older code and sharing the file keep working. A paper one of them re-indexes counts as stale again.
• has() is false for a stale paper, so it is re-chunked the next time it is indexed. Passages whose text did not change keep their vectors, so only new text is embedded.
• status reports stale.

Tested: 287 tests pass, including the detector’s positive and negative cases, vector reuse, an index from before versions existed, and a row rewritten by older code. ruff and pyright are clean.

🤖 Generated with Claude Code

Problems

1. Works the library creates itself (`pmcid-PMC5519140`, `doi-10.1214%2F12-aoas565supp`) were rejected by `index`, `collect` and the MCP `index`/`collect` tools with “unrecognised identifier”. One bad id also failed a whole batch.
2. `paper-fetch index` with no arguments aborted on the first unparseable record, leaving later works unindexed.

Changes

• `ids.normalize` parses `doi-`/`pmcid-`/`pmid-`/`arxiv-` work ids, including percent-encoded DOIs. Behaviour for ordinary DOI / OpenAlex / PMID / PMCID inputs is unchanged and junk is still refused.
• `index_works` reads text from the catalogue row (no re-parse) and skips and reports unreadable records under `skipped`; the CLI prints each with its reason and a count.
• `collect` returns a `rejected` list instead of failing the batch.
• When OpenAlex has no record, `fetch` fills title, year, authors and DOI from Europe PMC (never fatal).
• Existing library records are not rewritten; already-stored metadata-less works are only enriched on a new `fetch`.

Tests

`tests/test_own_work_ids.py`: 12 of its 19 cases fail on main and all pass here; full suite 278 passed, ruff and pyright clean.

🤖 Generated with Claude Code

Four changes that came out of live runs, plus the first companion tool.

One model call with no answer costs that item, not the run (follow-up to #3)

• Reading a passage: it is dropped with the flag no_answer_from_model and counted in Limits.
• Classifying a paper: it is graded unclear for this run, and that is not cached.
• Planning, synthesis and critique still end the run, as does an unreachable model.

Reranking is the paper library’s

• The local-model reranker is removed. In a live run the 4B model spent 4.9 of 5.7 hours reranking.
• The library’s cross-encoder (PAPER_FETCH_RERANK_MODEL) ranks passages. The pipeline keeps that order and only caps passages per paper and per sub-question.
• Without the setting, passages are read in retrieval order, and the run notes say so.

Two bugs from that run

• Fetch ordering now puts papers reported as open access first and papers reported as closed last. All six fetches had gone to closed papers.
• An exact quote match that stops inside a word or number is widened to the whole token. A stored quote read “5.” where the passage said “5.9 mmHg”.

Companion tools (research_pipeline/companions.py)

• With a companion configured under companions in the gateway config, the plan’s schema gains a tools field restricted to the configured keys.
• Code calls each requested tool once, within set bounds, and prints the tool’s own records in the answer. No model writes that section, and it never becomes evidence.
• A companion that fails is reported and the run goes on. Offline runs call none.
• First companion: datasets, which speaks dataset-fetch’s MCP contract.
  • It searches every catalogue that takes a free-text query, with Zenodo restricted to dataset-typed records.
  • It reads the data cards, and counts documents-only records instead of listing them.
  • It orders the rest by the library’s relevance.
• Adding another companion is one registry entry.

Also

• A generic Upstream MCP client; PaperLibrary is now one use of it.
• The design doc notes the planned “own endpoint” backend for private runs.

Tested:

• 237 tests pass. ruff and pyright are clean.
• Live against the real services: the library reranker returned on-topic passages, and the dataset companion searched three catalogues and read ten cards in 14 s.

🤖 Generated with Claude Code

Preregistration in progress. Stops at the -prereg tag for the operator to hear predictions before any run.

🤖 Generated with Claude Code

Found by a live run. One verify call generated 16,000 tokens without ever closing its JSON answer, hit the 600 s wall-clock limit, and ended the whole run. It happened twice.

• Output cap: every structured call is capped by PIPELINE_MAX_OUTPUT_TOKENS (default 2048), so a runaway call stops in about a minute.
• Narrower error: ModelOutputError, a subclass of LLMError, marks one failed call: out of time or tokens, or not JSON. An unreachable server or a client that stopped answering still raises LLMError and ends the run.
• Failed checks: a verifier call that fails this way is recorded as CHECK_FAILED and never counts as support.
• Unverified claims: a claim that no verifier answered for gets VERIFIER_FAILED. It is removed from the answer and counted in Limits. A real verdict from another check still decides the claim.

Tested:

• 230 tests pass, including tests for each case. ruff and pyright are clean.
• The live re-run completed. The two runaway checks were capped and reported in Limits, and the answer was produced.

🤖 Generated with Claude Code

Phase 3 of splitting the work between the two repositories. paper-fetch now owns everything about finding and indexing papers (polarizetech/paper-fetch#2, #3, #4). This repository tells it the field and the concepts, then does everything after a passage is found: reading it, judging it, and checking the claims made from it.

Changes

• Search: search takes the domain’s discipline profile (paper-fetch’s profile with the same slug) and an optional collection (--collection, PIPELINE_COLLECTION, or research_start(collection=)). Searches are remembered, and the run log lists earlier searches on the same concept.
• Planning: the planner reads the profile’s indexed terms, measures and search guidance. If the library has no such profile, the run says so.
• Fetching: the library’s relevance() decides the fetch order, and the library indexes the fetched texts.
• Retrieval: passages come from the library’s retrieve(), with stable ids (<work>#p<ord>) and the paper’s metadata. Evidence.passage_id is now that string id.
• Removed: index.py, the embedding calls, the ONNX reranker, and the numpy dependency. The local-model reranker stays. Study-design notes move to notes.sqlite, keyed by the text hash.
• Domains: each keeps its evidence half (taxonomy, measures, critique rules, prompt rules). Anchors, sources and search guidance are now paper-fetch profiles, and DomainPolicy.profile names the profile to use. The domain template ships a profile.toml, and DOMAINS.md explains both halves.
• Gateway: the rag__ tools read and index through the paper library’s MCP session. Evidence ids keep the same format.

Tests

• 225 tests pass, and ruff and pyright are clean.
• The tests run against a real in-memory paper-fetch Library. It’s a test-only dependency, pinned to a commit that is now on paper-fetch’s main.
• The gateway tests run paper-fetch’s real MCP server as the upstream, with every proxy pointed at a closed port so nothing can reach the network.
• New tests check that the profile reaches the planner, that profile and collection reach search, fetch and retrieval, and that a missing profile is noted.

Live migration (already done locally)

• The old index (735 papers, 33,852 passages, bge-m3 vectors) was copied into paper-fetch’s index without re-embedding. Re-splitting six sampled texts reproduced their passages exactly, and the text hashes matched.
• The 272 cached study-design notes were copied into notes.sqlite.
• A round trip through the real gateway worked: hybrid retrieval, then a passing citation check.

🤖 Generated with Claude Code

Two small additions that retrieval-augmented-generation needs before it can drop its own index and embedding calls:

• Each passage’s paper now includes route and format, so an application’s run log can say where the quoted copy came from.
• New Library.relevance(targets, texts) and MCP tool relevance: scores each text (for example a hit’s title) against the nearest target (for example a sub-question), from 0 to 1. It uses the configured embedder, or concept overlap without one.

Tests: 259 pass offline, and ruff and pyright are clean.

🤖 Generated with Claude Code

Phase 2 of moving paper searching and indexing into paper-fetch. This PR is stacked on #2 (the base is search-profiles-collections), so merge #2 first.

What moves here from retrieval-augmented-generation

• The passage index (passages.py). It is a local SQLite file built from held full texts, with FTS5 (BM25) always and dense vectors when PAPER_FETCH_EMBED_MODEL is set, fused by reciprocal rank.
• Chunking and hidden-character cleaning are unchanged, so passage offsets and text_sha256 mean what they did in the RAG index. The paper row now also records how many hidden characters were removed (hidden_chars).
• Passage ids have the form <work>#p<ord> and survive rebuilds. sha identifies the passage text.
• The ONNX cross-encoder reranker (rerank.py), set with PAPER_FETCH_RERANK_MODEL.
• The embedder and the reranker are both deterministic: they return vectors and scores, never text. The model-based reranker and the prompt-injection scan stay in the application, because both concern a model reading the text.

API

• Library.retrieve(queries, identifiers=, collection=, limit, per_paper, rerank) indexes its scope on demand before retrieving.
• Also index_works, paper_passages, and the passages / reranker properties.
• New MCP tools: retrieve, passages, index. New CLI commands: paper-fetch index [--rebuild] and paper-fetch retrieve.
• New optional extras: retrieval (numpy) and rerank (onnxruntime, tokenizers, huggingface_hub). The core still depends only on pypdf and fonttools.
• Embedding requests go through the new Http.post_json, so the “only http.py and the store open URLs” guarantee still holds.

Tested:

• 257 tests pass offline: exact offsets, stable ids across rebuilds, hybrid and lexical ranking, scope, the per-paper cap, reranking, refusing a different embedding model, and the MCP and CLI commands. ruff and pyright are clean.
• A live run with bge-m3 against the real library, writing the index to a scratch file, turned 15 held papers into 1,405 passages in 94 s. A repeat query took about 10 ms.

Next (phase 3): retrieval-augmented-generation drops its own index and reranker and calls these tools, and its domains keep only their evidence rules.

🤖 Generated with Claude Code