Problem
The global biochar research community has exchanged practical and scientific knowledge through email lists since 2008. Five subgroups accumulated more than 42,000 raw messages plus research papers, microscope images, data sheets, and field reports. This is an independent community project, not client work.
Keyword search could locate a term but could not synthesize evidence across long conversations or place related images and PDFs beside the relevant discussion. The system needed to make the archive searchable by meaning, preserve source context, support multilingual questions, and operate at a cost appropriate for a community resource.
The first version exposed a corpus-level dependency. When its embedding provider withdrew the selected model, recovery required re-embedding the archive. The rebuilt system keeps source artifacts, checkpointed build stages, compact runtime artifacts, and dimension checks so model lifecycle risk is visible and recoverable.
The counts below describe different pipeline stages and should not be treated as interchangeable. The source pass extracted 41,586 email messages and 1,465 attachments. Cleaning and record construction produced 42,083 searchable records, which then generated 238,078 text chunks and 910 media vectors.
Architecture
Data ingestion pipeline
The system processes a consolidated 4.5 GB MBOX file through a five-stage pipeline:
1,465 attachments
Processed
1,000 / 200 chars
768 dimensions
191 PDFs
Extraction and normalization parse message metadata and body text, retain conversation structure from email headers, deduplicate attachments by hash, strip non-content markup, and record language. Each downstream artifact remains linked to its source record.
Chunking splits clean text into 1,000-character segments with 200-character overlap. Each chunk retains parent metadata so a retrieved passage can be traced back to the original message and thread.
Embedding maps text, images, and PDF pages into one 768-dimensional space. The batch pipeline uses adaptive rate control and checkpoints every 5,000 items, allowing a long build to resume without restarting the corpus.
Runtime compaction converts the full-precision index into an 8-bit scalar-quantized FAISS inner-product index. The September 2026 runtime holds all 238,988 vectors in a 183.5 MB index, down from approximately 734 MB at full precision. Recorded validation produced 0.9918 recall at 20 against the stored-vector flat-index baseline, with identical top-1 results.
Two-pass retrieval
In system testing, text-to-text similarity scores ran 0.2 to 0.4 higher than text-to-image scores. A naive top-k search therefore buried media results under text matches even when an image or PDF page was relevant.
The solution is a two-pass strategy. Pass 1 runs standard FAISS search across all 238,988 vectors. Pass 2 runs a dedicated search over only the 910 multimodal vectors, reserving the top 5 results above a 0.20 similarity threshold. Text results fill remaining slots, and the combined list is re-sorted and re-ranked.
Before this intervention, the test queries returned no media results in the top 20. The reserved-slot method made the 719 image vectors and 191 PDF-page vectors operationally useful without maintaining a second user-facing search path.
Query processing
Each question passes through language normalization, 768-dimensional query embedding, two-pass retrieval, and answer generation with source references. Generation model selection is configurable; the retrieval index remains tied to its corpus embedding space and is managed as a separate lifecycle dependency.
Infrastructure
| Component | Technology |
|---|---|
| Vector store | FAISS 8-bit scalar-quantized inner-product index, 183.5 MB |
| Corpus store | SQLite, fetching chunk text and metadata by row position |
| API | FastAPI (uvicorn) |
| Frontend | Streamlit with inline image display |
| AI services | Vertex AI multimodal embeddings and configurable Gemini generation |
| Hosting | GCP Compute Engine e2-micro, 1 GB RAM, non-preemptible |
| Reverse proxy | Caddy with automatic TLS |
| Process control | systemd with restart on crash or reboot |
| Monitoring | Five-minute health checks, application restart path, and operator alerts |
Why this architecture
A shared multimodal space avoids separate text and image indexes plus a late-fusion layer. One search can compare a question with email passages, images, and PDF pages. The dedicated media pass then corrects the score imbalance observed between modalities.
A compact runtime matters because the community workload does not justify a large always-on machine. Quantization, on-demand SQLite reads, and separation of API and interface memory reduced the production footprint enough to run on a 1 GB VM.
Lifecycle controls matter more than a model name. The source archive, full build, compact runtime set, code, hashes, and manifest are retained separately. Model changes still require re-embedding, but the rebuild path is explicit and resumable.
Evidence boundary: the retrieval recall result compares the compact index with the full-precision index over stored vectors. It measures quantization loss, not answer correctness or scientific validity. Source citations let users inspect the underlying community material.
Performance
The quantized vector scan completes well under 100 milliseconds in the current runtime. End-to-end response time depends primarily on answer generation, not retrieval. Chunk text remains in SQLite and is fetched only for selected results, keeping the full corpus out of API memory.
At approximately 500 questions per month, the observed September 2026 run rate is about $5 to $6 per month. The current basis includes an in-use external IP, a retained rollback image, and metered AI calls; compute, disk, scheduler, and health-function usage fall within applicable free tiers. Cost changes with traffic, model pricing, storage, and free-tier eligibility.