Narrows a daily stream of new papers down to the few worth reading, then pulls grounded evidence out of those few. It is a screening system, not a question-answering system over everything.
That distinction was not the starting assumption. It is what the retrieval evaluation forced.
| Capability | Entry point |
|---|---|
| External source monitoring, filtering, daily brief | application/external_monitoring.py |
| PDF → structured chunks via MinerU | parsers/mineru_adapter.py |
| Hybrid retrieval — dense + sparse RRF, lexical backfill, selective rerank | store/qdrant_store.py |
| Structured query-rewrite routes (entity / table_evidence / alias) | retrieval/query_rewrite.py · retrieval/multi_query.py |
| Evidence-grounded agent — answers must carry structured citations | agent/literature_agent.py |
| Retrieval evaluation harness — the reason any of the above is decidable | evals/retrieval_eval.py |
Read the two series against the job. Paper-level hit — did the right paper surface at all — reaches 0.983 with cheap lexical backfill. Chunk-level hit — did the exact evidence passage surface — tops out at 0.667 while latency triples.
An earlier top-100 diagnostic explains why: chunk_hit@100 is only 0.500, so the target chunk is
frequently absent from the candidate pool entirely. No reranker recovers what was never
retrieved. The cause is corpus structure — in a body of papers repeating one methods vocabulary,
more documents make chunk-level discrimination harder.
So the product is scoped to screening, and evidence extraction is a reading aid within a small selected set.
git clone https://github.com/77652189/SortPaper.git
cd SortPaper
pip install -r requirements.txt
cp .env.example .envpython -m streamlit run app.pyQdrant and the parser start on demand (ADR-0024) — online acquisition and the daily brief run without either. That is deliberate: screening is the part that runs every day, so it must not depend on the parts that run occasionally.
The monitor is a Windows scheduled task, not a background thread of the Streamlit process (ADR-0016):
./install_papersort_external_monitor_task.batIt keeps fetching with the browser closed. scripts/papersort_control.ps1 starts, stops and
diagnoses the services; scripts/start_papersort_lan.ps1 exposes the UI on the LAN.
To run fully offline with the local Qwen3 provider (ADR-0032):
./tools/start_qwen3_embedding_server.ps1Switching provider is not a drop-in — a changed model or dimension requires a rebuilt collection (ADR-0007).
| Layer | Choice | Why this one |
|---|---|---|
| Vector store | Qdrant | Dense and sparse vectors in one collection, so hybrid retrieval and RRF happen server-side rather than being stitched in the app (ADR-0006) |
| PDF parsing | MinerU | Handles three-column layouts and borderless tables out of the box — the case a self-built pipeline had been carrying (ADR-0003) |
| Embeddings | OpenAI · DashScope · local Qwen3 via sentence-transformers | Provider is abstracted; the local path exists so the pipeline can run with no external API at all |
| Rerank | DashScope qwen3-rerank, triggered selectively |
Measured: selective and full rerank scored identically, and selective was faster (ADR-0033) |
| UI | Streamlit | Single-user local research tool; workspaces organised by task, not by module (ADR-0008) |
| Scheduling | Windows scheduled task | Monitoring must survive the UI being closed, so it cannot live inside the UI process |
| Tests | pytest | 69 test files; the retrieval eval harness is separate and does not auto-change main-path defaults |
langgraph and faiss-cpu are still in requirements.txt — they belong to the superseded parsing
path described below, kept reachable for regression comparison. They are not the current
architecture.
Dependencies point inward. Domain rules may not import Streamlit, the Qdrant SDK, an LLM SDK, Docker, or PowerShell; the application layer reaches outward only through ports.
flowchart TB
UI["Streamlit UI<br/>app_workbench · app_external_ui"]
APP["application/ — use cases<br/>paper_pipeline · agent_search · external_*"]
DOM["domain/ — models & policies"]
PORTS["ports/ — contracts"]
ADP["adapters/ — MinerU · Qdrant · LLM · sources · Windows runtime"]
UI --> APP --> DOM
APP --> PORTS --> ADP
Three data chains, deliberately not merged:
| Chain | Path | Rule |
|---|---|---|
| Paper metadata | sources → filter → classify → translate → JSON repository → candidate library, daily brief | must not depend on Qdrant |
| PDF & retrieval | PDF → LayoutChunk → verdict/quality → Qdrant → retrieval / agent | MinerU is the primary parser |
| Runtime services | use case → RuntimeServicePort → Docker / local model / process control |
the UI never executes a process itself |
Keeping the metadata chain free of Qdrant is what lets screening still run when the vector store is down — and screening is the part that runs every day.
Academic tables are the hardest part of the corpus, and booktabs tables are the hardest of those: no vertical rules by definition, so cell-boundary detection has nothing to detect — compounded by two-column layouts and rotated placement. At the time, open-source parsers did not handle this case badly; they skipped it. There was nothing to adopt.
The first approach — fixing documents one at a time — was correct and did not scale.
What made it scale was turning an LLM judge from a scorer into a source of structured corrective actions:
Both gates matter. Without the retry cap the loop runs away; without the margin it oscillates between two equally mediocre parses.
The residual failure modes then traded off against each other: tighten the thresholds and tables duplicate less but go missing more. The duplicating configuration was chosen, because the two errors are not symmetric — a duplicate is visible to the reader and removable downstream, while a missing table is silent and may be the exact evidence being looked for. The code still shows the asymmetry: the retry path handles false positives, and for missing content there is no second parser to fall back on. ADR-0005 fixes the same principle architecturally — suspicious candidates keep their diagnostics and are not deleted by any single step.
Then MinerU handled the same cases off the shelf, and
ADR-0003 made it the default. The superseded LangGraph
pipeline is still in the repository, reachable only through
legacy/parsing.py, kept as regression evidence rather than architecture.
The lesson is not "look for a library first" — at the start there was nothing to find. It is that a technology survey has to be time-boxed against the build cost and repeated. The first survey was right; the mistake was never running it again over the following months, in a period when that tooling moved quickly. A one-time survey expires whenever the project outlives the tools' iteration cycle.
- Not a general QA system over the whole corpus. Screen first, extract evidence within a small selected set second.
- Chunk-level recall is ~0.67 at best. Extraction is a reading aid, not an authority — every agent answer carries citations so the claim can be checked (ADR-0019).
- The legacy parse path is not maintained, compatibility and regression only.
- Changing embedding model or dimension requires a rebuild. Not a config toggle.
- Windows-oriented runtime. Docker and local-model control live in the runtime adapter; the UI cannot execute processes (ADR-0025).
- Evaluation uses weak labels auto-generated from the indexed corpus, with hard-case queries built from the target chunk's own keywords. Good for regression and direction; not an absolute quality claim.
| Document | Changes when |
|---|---|
| Requirements | the goal or capability boundary changes |
| Architecture | the implementation structure changes |
| Execution plan | progress moves — sole authority on status |
| Handoff | the active slice changes |
| ADR index | never — decisions are superseded, not edited |
More work at my personal site.