Discipline profiles, named collections and search memory
Phase 1 of moving paper searching and indexing out of retrieval-augmented-generation and into paper-fetch. Nothing here uses a language model: everything is string matching and bookkeeping, so it is repeatable and tested offline.
Discipline profiles (profiles.py, data in profile_data/*.toml)
- A profile holds the search half of a field: controlled terms (MeSH ids are validated), synonyms, measures, sources, and search guidance for whoever writes the queries.
search(profile=)also runs up to two variants that swap a synonym for its indexed term (“HRV” → “heart rate variability”). It never swaps the other way. Hits whose titles name the field’s terms rank first (profile_match), and results are merged across variants (found_by).- Four profiles ship built in. They were converted from the RAG repo’s domains. They don’t narrow the provider set, because that would drop the repository providers that find open copies.
- A TOML file in
~/.config/paper-fetch/profiles/adds a profile or replaces a built-in one by slug.
Named collections (collection.py)
papers/collections/<name>.jsonholds a member list and a default profile. A paper is stored once, whichever collections list it.fetch(collection=)lists the paper in the collection.collectanduncollectadd or remove members by any identifier.
Search memory (memory.py)
papers/memory/searches.jsonlgets one row per search, written with the same re-read-and-append approach as the catalogue.- Every search reports up to three earlier searches on the same concept, and marks each hit that an earlier search already found (
seen_before). recalllooks up past searches by concept, profile, collection or paper. A concept is the set of content words with synonyms mapped to indexed terms.
Surfaces
- New MCP tools:
profiles,recall,collections,collection,create_collection,collect,uncollect.searchandfetchtakeprofileandcollection. - New CLI commands:
profiles,collection,collect,recall, plus--profileand--collectiononsearchandfetch.
Tested:
- 239 tests pass offline, including concurrent writers to the memory and to a collection. ruff and pyright are clean.
- A live search with the cardiovascular profile, run on an in-memory store, found the “heart rate variability” variant hits, ranked them first, and recalled the earlier “HRV” search at a score of 0.8.
Next:
- Phase 2: the passage index and retrieval move here.
- Phase 3: retrieval-augmented-generation calls paper-fetch for search, memory and retrieval, and its domains keep only the evidence half.
🤖 Generated with Claude Code