Work · Repository

retrieval-augmented-generation

Evidence-first literature RAG for small local models: code controls search, evidence, verification and the run log

Python
MIT licence
1 post
9 merged PRs
1 star

Posts about it

2026-09-24 A RAG tool that keeps the model in check A short, evidence-first protocol for using retrieval-augmented generation without letting a language model turn an unsupported answer into a finding. [A]

README

Evidence-first literature research with small local language models. Code controls search, evidence, verification, abstention and the run log; the model only fills in narrow, schema-constrained forms. Every claim in an answer cites a verbatim passage from a full-text paper, and claims that fail a check are removed and counted rather than softened.

The method is described in A RAG protocol that makes the model show its work. The design document gives the rationale, the literature behind each rule, and a table mapping every rule to the code that enforces it and the test that checks it.

[!IMPORTANT] This does not make a language model truthful. It narrows the paths by which unsupported claims reach an answer and makes the remaining uncertainty visible. Read the known limits before relying on an answer.

What it does

There are two ways to use it:

The pipeline (research-pipeline ask) answers a question end to end with a local model: plan, search, fetch open-access full text, retrieve and rerank passages, extract evidence, synthesise claims, search again for null results, verify, grade, and write answer.md plus a run.json that records every search, passage, model digest and verdict.
The evidence tools (rag__* over MCP) let a stronger client model (Claude, ChatGPT) write the answer while this repository holds it to the same deterministic rules: evidence only from the paper library’s passage index, quotes that must occur in the cited passage, and numbers that must occur in the cited evidence. These tools do not judge entailment; the pipeline’s verifier models do.
The quantity tools (math__* over MCP) give a client model constants and equations to cite instead of recall: math__lookup searches the research corpus’s calculator records (projects/*/calculators/*.md, found through $RESEARCH_CORPUS or a sibling research/ checkout) and returns their sections verbatim with the corpus commit; math__bionumber returns a BioNumbers entry by BNID, verbatim, fetched once and cached. No model is involved in either.

Finding and indexing papers is not done here. That is the paper library, paper-fetch, reached over MCP: it searches the open-access providers, knows how each scientific discipline is indexed (a profile: its MeSH terms and synonyms), remembers past searches, keeps project collections, stores legal full text, and owns the passage index that retrieval reads. This repository tells it the field and the concepts, then does everything that happens after a passage is found: reading it, judging it, and checking every claim made from it.

paper-fetch (search and indexing)	this repository (evidence)
provider search, discipline profiles, search memory, collections	question planning (with the profile’s vocabulary)
open-access fetching, storage, provenance	extraction, study-design classification
passage index: chunking, BM25, embeddings, optional cross-encoder	synthesis, verification, grading, critique
fetch ordering by relevance to sub-questions	quote, number and retraction checks; the answer and run log

What each rule guarantees, and what it does not:

Rule	Enforced by
A claim can only cite evidence retrieved in this run	synthesis schema enum, then checked in code
A quote must occur in the stored passage	`verify.anchor_quote`: exact match, or a near-identical repair that may not change a negation or a number
A number in a claim must occur in its cited evidence, with a matching unit where the field declares units	`verify.unsupported_numbers`; a failing claim is removed
Verifier disagreement is shown, not averaged	`verify.settle` → `DISPUTED`
A passage describing another paper is a pointer, not evidence	model judgement plus a citation-marker heuristic that can only demote
Retrieved text addressing an AI reader is excluded	an English pattern list: catches careless cases, not a determined adversary
Retracted sources support nothing	OpenAlex flag at fetch, Crossref/Retraction Watch at answer time
The bibliography is printed from stored records	`render.py`; the model never types an author, year or DOI

Requirements

Python 3.11+ and uv
The paper library, paper-fetch, over MCP. Install it with its mcp and retrieval extras, and give it an embedding model for hybrid retrieval (PAPER_FETCH_EMBED_MODEL=bge-m3, about 1.2 GB in Ollama); without one, retrieval is lexical.
For local-model runs, Ollama with a text model (default qwen3:4b-instruct-2507, about 2.5 GB). Sized for a machine with 16 GB of memory running one model at a time. Behind MCP the calling model does this work instead.

Install

git clone https://github.com/polarizetech/retrieval-augmented-generation.git
cd retrieval-augmented-generation
uv sync
ollama pull qwen3:4b-instruct-2507
cp config/rag.env.example config/rag.env
cp config/mcp-gateway.example.json config/mcp-gateway.json

uv tool install "paper-fetch[mcp,retrieval] @ git+https://github.com/polarizetech/paper-fetch"
ollama pull bge-m3
echo "PAPER_FETCH_EMBED_MODEL=bge-m3" >> ~/.config/paper-fetch/retrieval.env

paper-fetch-mcp must be on your PATH, or its absolute path in config/mcp-gateway.json. Reranking is paper-fetch’s cross-encoder: install paper-fetch[rerank] and set PAPER_FETCH_RERANK_MODEL (see its README); without it passages are read in retrieval order.

Use

uv run research-pipeline index        # have the library index the full texts it holds
uv run research-pipeline ask "Does exercise training lower resting blood pressure in adults?"
uv run research-pipeline ask --domain cardiovascular "..."   # the field's rules and vocabulary
uv run research-pipeline ask --collection my-review "..."    # search, fetch and read in a collection
uv run research-pipeline ask --offline "..."   # indexed corpus only: no search, no downloads
uv run research-pipeline status

Each run writes runs/<timestamp>-<slug>/answer.md and run.json. An online run fetches open-access papers into the paper library; PIPELINE_MAX_FETCH bounds how many. Every search is remembered by the library, so a later run on the same concept is told what earlier ones found.

Set PIPELINE_VERIFIER_MODEL to a model from a different family than the text model. Without it the text model checks its own claims, and every answer says so.

Companion tools: datasets and patents

A run can loop in another tool when the question calls for it. The planner is told what each configured companion is for and may request it, with queries, in the plan it already writes; code then calls the tool and prints what its own records say. No model writes that section, and it never counts as evidence for a claim.

Two companions ship. Both are off until enabled under companions in config/mcp-gateway.json, and each is asked only when the planner judges that the question calls for it; most questions call for neither.

datasets asks dataset-fetch’s recommend (it needs a dataset-fetch with that tool) for datasets that could test the question, and adds a “Datasets that could test this” section: title, catalogue, licence, files, subjects, DOI, a pinned reference and why it was listed. Nothing is downloaded or assessed.
patents runs patent-fetch’s patent-fetch --json search when the question is about a device, a method or another invention, and adds an “Existing patents on this” section: title, publication number, year, applicants or inventors, classes and a link, one entry per invention: the titles closest to the question, then the most recent of the rest. Nothing is fetched or read, and it is not a legal opinion: the section says so, and the limits name every patent service that was not asked or did not answer. Without a patent service account, patent-fetch can only search life-science patents up to 2012; the answer says that too.

Adding another tool is one entry in research_pipeline/companions.py: an MCP server or a command line that prints JSON.

Novelty probe: has someone already done this?

uv run research-pipeline novelty --id CAND-0007 "The candidate claim, written out in full" \
    --established "the closest established terminology" \
    --queries "phrasing A" "phrasing B" "phrasing C" "phrasing D" \
    --research-dir ../research        # or set RESEARCH_REPO

An adversarial prior-art search. Its null hypothesis is that the claim is already published, and it tries to prove that:

It searches every index the paper library federates, once per phrasing. At least five phrasings are required, one of them in the field’s established terms.
It fetches the nearest open-access papers in full.
It puts each retrieved passage to the verifier with the candidate as the claim.

A search hit is not prior art; a passage that states the claim is. There are four verdicts:

Verdict	Meaning
`PRIOR_ART`	Every verifier accepts a passage, its quote is found in the stored text, and every number in the claim is present in it.
`PARTLY_KNOWN`	A passage states a weaker or narrower version, or the verifiers disagree.
`INCONCLUSIVE`	An index refused, the run was offline, or fewer than three of the nearest works could be read.
`CANDIDATE`	None of the above. It describes only the works read, not the literature.

The dossier (prior-art.md, plus the full run log under runs/) is written into the research repository at novelty/<ID>/, next to the corpus it concerns. The command refuses any directory that does not carry that repository’s marker files. The dossier never announces a discovery. It does quote the statement as given, and reports what the earlier any-hit rule would have said, alongside what the reading found.

As an MCP server

For Claude Desktop or Claude Code, run the gateway over stdio (no port, no token):

{
  "mcpServers": {
    "research-rag": {
      "command": "/absolute/path/to/retrieval-augmented-generation/.venv/bin/research-rag-gateway",
      "args": ["--config", "/absolute/path/to/retrieval-augmented-generation/config/mcp-gateway.json"]
    }
  }
}

It exposes rag__search, rag__retrieve_evidence, rag__check_citations, rag__save_report, the paper library as papers__*, and the whole pipeline as pipeline__*. Through MCP the pipeline uses the calling model (Claude, ChatGPT) for its model steps by default: code runs every stage and hands the client batches of tasks with a JSON schema to answer, and Ollama is only needed for embeddings. See MCP clients for the workflow and the HTTP transport.

Domains

The engine is domain-neutral. What a field knows (which study designs count and in what order, which databases are authoritative, what a unit means, what may not be generalised) lives in a separately versioned package under domains/, discovered through an entry point. The engine never imports one; a policy is data and is written into run.json.

uv pip install -e domains/cardiovascular
uv run research-pipeline ask --domain cardiovascular "Does higher baroreflex sensitivity predict lower cardiovascular mortality?"

Domain	What it adds
`cardiovascular`	Cardiac, vascular and autonomic evidence; a surrogate is not an outcome.
`respiratory`	Breathing, gas exchange and the chemoreflex; lung function is not a clinical endpoint.
`vestibular`	Balance and dizziness, graded by diagnostic accuracy rather than by trial.
`neuroscience`	The nervous system, with the species gap and reverse inference as failure modes.

Writing your own: domains.

Evaluation

uv run python benchmarks/research/verifier_bench.py   # does the verifier reject what it should?
uv run python benchmarks/research/eval_pipeline.py    # end-to-end cases, scored from run logs

Both are scored without an LLM judge. The seed sets are small (7 verifier pairs, 4 questions): they are smoke tests and templates, not accuracy estimates. See benchmarking.

Development

uv sync --group dev
uv run pytest --cov          # offline: no Ollama, no network
uv run ruff check . && uv run ruff format --check .
uv run pyright

CI runs the same checks on Python 3.11 to 3.13.

Security

Ollama and the gateway’s HTTP transport have no meaningful authentication. Keep them on loopback (scripts/mcp-start.sh refuses any other bind unless MCP_ALLOW_PUBLIC_BIND=1 is set) and put an authenticating proxy in front if remote access is needed. Retrieved papers are untrusted input: the pipeline wraps them as data and holds no write tool except the library’s own fetch.

License

MIT

Changelog

2026-10-13 #9

Document how to add a paid patent service for the pipeline alone

Documentation only; no code changes.

patent-fetch asks its paid providers (SerpApi) only when they are named. The patents companion’s config entry already passes an env to the command it starts, so naming the provider set there adds one for the pipeline alone:

"env": {"PATENT_FETCH_SEARCH_PROVIDERS": "europepmc,epo-ops,lens,serpapi"}

A search from patent-fetch’s own command line still asks its usual set.

config/mcp-gateway.example.json: an empty env on the patents companion and an _env_comment saying what it is for, what it needs, and what it costs (one paid search per planner query, up to seven more when a query matches nothing and is retried).
docs/research-pipeline.md: a paragraph on the same, with what a live check on 2026-10-13 returned. The patents paragraph above it is rewrapped; it had lines over 100 columns and its text is unchanged.

Checked: the example config still parses and the companion still loads from it; 252 tests pass.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-13 #8

Patents companion: short queries, a fallback for empty ones, room for recent patents

The “Existing patents on this” section of a run on 2026-10-13 showed only patents from 1996 to 2001, although the patent services hold ones from 2024 and 2025 on the same subject.

Cause. A patent service returns only records with every word of a query in the title or abstract. The planner had been asked for queries “naming the invention and what it does the way a patent title would”, and wrote a six-word one that matched nothing at either service; the other matched only patents using an older term.

Changes, all in research_pipeline/companions.py:

The planner is asked for 1-3 queries of 2 to 4 words each, and told that every extra word finds fewer patents (max_queries 2 → 3).
A query of 3 to 7 words that matches nothing is asked again with one word left out at a time. The answer’s limits name the query, say why it matched nothing, and say how many records asking again found.
Of the six entries listed, four are the closest titles and two are the most recent of the rest, so old patents whose titles repeat the question cannot fill every place. The section’s first line says this.

Checked against the live services with the two queries from that run: the six-word query now recovers 10 records through the fallback, and the section lists patents from 2025, 2024, 2016 and 2014 beside those from 1999 and 2011. Tests: 252 pass; ruff, pyright clean.

Not checked: a full pipeline run with a model writing queries to the new instruction.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-13 #7

Patents companion: list an invention once, with its other publications

patent-fetch now folds the publications of one invention (an application, its grant, the same filing at other offices) into one hit and names the others in also_published_as (polarizetech/patent-fetch@9552d1f).

This makes the patents companion use that:

Two planner queries can each keep a different publication of the same invention, so a hit counts as already listed when its number, or any number it was also published under, was listed before.
The “Existing patents on this” section prints Also published as ... after an entry (four numbers, then a count).

Why: a pipeline run on 2026-10-13 listed five patents, of which four were the US, EP, WO and CA publications of one Cochlear patent. The same two queries now list two inventions.

Older patent-fetch versions send no also_published_as; the companion then behaves as before.

Checked: ruff, pyright, 250 tests; and the companion run against the live patent services with the queries from that run.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-07 #6

Patents companion: the plan may ask patent-fetch what is already patented

Adds a second companion tool, built the same way as datasets. It is off until enabled, and is asked only when the planner judges that the question is about a device, a method or another invention.

A companion can now be a command

patent-fetch ships a command line and no MCP server, and offers patent-fetch --json search for gateways.
upstream.Command runs a configured command and reads its standard output as JSON.
The command is started with an argument list, never through a shell, so a query a model wrote is always one argument.
It reports a missing command or a timeout as unavailable, and a non-zero exit with the last line of stderr.
Companion.transport (mcp or command) picks the client.

The patents companion

Runs one search per planner query and merges hits by publication number.
patent-fetch does not rank, so the hits are ordered by the paper library’s relevance of each title to the question.
Prints the services’ own records under “Existing patents on this”: number, year, applicants or inventors, classes, status in the service’s words, and a link.
Fetches nothing, and never becomes evidence for a claim.
The section states that it is not a legal opinion and that finding none is not evidence that none exists.
Limits name every service that had no account or did not answer. They also say when only Europe PMC’s archive answered (life-science patents, none after 2012), so a search that could not see recent patents does not read as “none exist”.

Config (config/mcp-gateway.example.json): "patents": {"command": "patent-fetch", "args": ["--json"], "enabled": false}.

Tested:

249 tests pass, and ruff and pyright are clean.
New tests cover merging and ordering, each limits case, and the command transport: a shell-hostile query passed as one argument, a non-zero exit, non-JSON output, a timeout, and a command that is not installed.
New tests also cover a full run that requests patents, and a run where the tool is missing and the run continues.
A live check against the real command returned 16 records for two queries, listed six, and gave the correct limits.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-03 #5

Datasets companion: one call to dataset-fetch's recommend

dataset-fetch now has recommend(topic) (polarizetech/dataset-fetch#12). Choosing catalogues, phrasing queries and ranking datasets now live there, the way paper search lives in paper-fetch. This switches the datasets companion to that one call.

Calls: one recommend call per planner query, at most two.
Merging: cards are merged by reference, keeping the better score. Records with verified data files come first, then each by dataset-fetch’s own score.
Printing: the answer shows the tool’s own fields: catalogue and kind, licence, files, subjects, modalities, creators, DOI, a pinned reference, and the tool’s reason for listing it. Records that hold only documents, and lower-ranked records, are counted rather than listed.
Failures: a query that recommend refuses doesn’t discard the other query’s results. A catalogue that was silent on every query is named in Limits.
Removed: the companion’s own loop over catalogues, the Zenodo query rewrite, the card reading and the document filter.

Tested:

238 tests pass, and ruff and pyright are clean.
Live: one query, about 19 s.
- dataset-fetch asked OpenNeuro, PhysioNet, Zenodo and OSF, and skipped the other ten with reasons.
- Every listed record held data, including PhysioNet blood-pressure databases and a structured-exercise dataset.
- The old loop listed none of six.

Needs a dataset-fetch installed with recommend.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-02 #4

Library reranking, per-item failures, fetch and quote fixes, companion tools

Four changes that came out of live runs, plus the first companion tool.

One model call with no answer costs that item, not the run (follow-up to #3)

Reading a passage: it is dropped with the flag no_answer_from_model and counted in Limits.
Classifying a paper: it is graded unclear for this run, and that is not cached.
Planning, synthesis and critique still end the run, as does an unreachable model.

Reranking is the paper library’s

The local-model reranker is removed. In a live run the 4B model spent 4.9 of 5.7 hours reranking.
The library’s cross-encoder (PAPER_FETCH_RERANK_MODEL) ranks passages. The pipeline keeps that order and only caps passages per paper and per sub-question.
Without the setting, passages are read in retrieval order, and the run notes say so.

Two bugs from that run

Fetch ordering now puts papers reported as open access first and papers reported as closed last. All six fetches had gone to closed papers.
An exact quote match that stops inside a word or number is widened to the whole token. A stored quote read “5.” where the passage said “5.9 mmHg”.

Companion tools (research_pipeline/companions.py)

With a companion configured under companions in the gateway config, the plan’s schema gains a tools field restricted to the configured keys.
Code calls each requested tool once, within set bounds, and prints the tool’s own records in the answer. No model writes that section, and it never becomes evidence.
A companion that fails is reported and the run goes on. Offline runs call none.
First companion: datasets, which speaks dataset-fetch’s MCP contract.
- It searches every catalogue that takes a free-text query, with Zenodo restricted to dataset-typed records.
- It reads the data cards, and counts documents-only records instead of listing them.
- It orders the rest by the library’s relevance.
Adding another companion is one registry entry.

Also

A generic Upstream MCP client; PaperLibrary is now one use of it.
The design doc notes the planned “own endpoint” backend for private runs.

Tested:

237 tests pass. ruff and pyright are clean.
Live against the real services: the library reranker returned on-topic passages, and the dataset companion searched three catalogues and read ten cards in 14 s.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-10-01 #3

A verifier call with no answer fails that check, not the run

Found by a live run. One verify call generated 16,000 tokens without ever closing its JSON answer, hit the 600 s wall-clock limit, and ended the whole run. It happened twice.

Output cap: every structured call is capped by PIPELINE_MAX_OUTPUT_TOKENS (default 2048), so a runaway call stops in about a minute.
Narrower error: ModelOutputError, a subclass of LLMError, marks one failed call: out of time or tokens, or not JSON. An unreachable server or a client that stopped answering still raises LLMError and ends the run.
Failed checks: a verifier call that fails this way is recorded as CHECK_FAILED and never counts as support.
Unverified claims: a claim that no verifier answered for gets VERIFIER_FAILED. It is removed from the answer and counted in Limits. A real verdict from another check still decides the claim.

Tested:

230 tests pass, including tests for each case. ruff and pyright are clean.
The live re-run completed. The two runaway checks were capped and reported in Limits, and the answer was produced.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-09-30 #2

Hand searching and indexing to paper-fetch

Phase 3 of splitting the work between the two repositories. paper-fetch now owns everything about finding and indexing papers (polarizetech/paper-fetch#2, #3, #4). This repository tells it the field and the concepts, then does everything after a passage is found: reading it, judging it, and checking the claims made from it.

Changes

Search: search takes the domain’s discipline profile (paper-fetch’s profile with the same slug) and an optional collection (--collection, PIPELINE_COLLECTION, or research_start(collection=)). Searches are remembered, and the run log lists earlier searches on the same concept.
Planning: the planner reads the profile’s indexed terms, measures and search guidance. If the library has no such profile, the run says so.
Fetching: the library’s relevance() decides the fetch order, and the library indexes the fetched texts.
Retrieval: passages come from the library’s retrieve(), with stable ids (<work>#p<ord>) and the paper’s metadata. Evidence.passage_id is now that string id.
Removed: index.py, the embedding calls, the ONNX reranker, and the numpy dependency. The local-model reranker stays. Study-design notes move to notes.sqlite, keyed by the text hash.
Domains: each keeps its evidence half (taxonomy, measures, critique rules, prompt rules). Anchors, sources and search guidance are now paper-fetch profiles, and DomainPolicy.profile names the profile to use. The domain template ships a profile.toml, and DOMAINS.md explains both halves.
Gateway: the rag__ tools read and index through the paper library’s MCP session. Evidence ids keep the same format.

Tests

225 tests pass, and ruff and pyright are clean.
The tests run against a real in-memory paper-fetch Library. It’s a test-only dependency, pinned to a commit that is now on paper-fetch’s main.
The gateway tests run paper-fetch’s real MCP server as the upstream, with every proxy pointed at a closed port so nothing can reach the network.
New tests check that the profile reaches the planner, that profile and collection reach search, fetch and retrieval, and that a missing profile is noted.

Live migration (already done locally)

The old index (735 papers, 33,852 passages, bge-m3 vectors) was copied into paper-fetch’s index without re-embedding. Re-splitting six sampled texts reproduced their passages exactly, and the text hashes matched.
The 272 cached study-design notes were copied into notes.sqlite.
A round trip through the real gateway worked: hybrid retrieval, then a passing citation check.

🤖 Generated with Claude Code

Pull request on GitHub →

2026-09-29 #1

Gateway: a server section names an instance and can drop the built-in tools

One gateway binary can now front a different set of upstreams under its own name. A second instance (the org-status MCP scientific-research-news) can then reuse the OAuth server instead of copying oauth.py.

New optional server section in the gateway config: name, instructions, and builtin_tools. With builtin_tools: false the instance drops web__search and the rag__* tools, so it lists only gateway_status and its upstreams.
Omitting the section keeps today’s behaviour exactly.
Tests: one new test in tests/test_gateway.py; 9/9 pass. Ruff is clean. Documented in docs/mcp-clients.md.

🤖 Generated with Claude Code

Pull request on GitHub →