Meeting Notes RAG

Update: gapvec is retired. The follow-on covers why, and it is not that retrieval fell short. The Rust rewrite lives at gapvec with its companion glean; the original Python implementation is preserved at gapvec-py.

A local RAG pipeline for searching customer meeting notes spanning multiple years, benchmarked against Glean and Gemini. The local system reaches 81% of Glean’s score (3.37 vs 4.15) running fully offline on Apple Silicon. The gap is largely a function of input scope: the local system deliberately indexes a single document per account while Glean indexes 20+ sources across 8 platforms. Within that constraint, most of the remaining gap is a retrieval problem fixable with techniques that have nothing to do with embeddings.

The Corpus

I’m a Customer Success Manager at GitLab managing accounts in highly regulated industries (defense, intelligence, finance) where data sovereignty is sensitive. Most customers at this size and complexity are integrating GitLab into a complex ecosystem of tools, and their inquiries go far beyond explaining documentation. The bulk of engagement work requires scanning source, issues, and merge requests, then reasoning through deep integration questions. Each customer has a long-running document serving as the canonical record: cadence calls, architecture sessions, escalations, follow-ups. Some accounts have accumulated 200-400 pages of notes over their lifetime. The richness is the value: who said what, when a topic surfaced, which version was running when something broke.

Before building this, I was confronted with a tab and document management challenge. The most progressive option at the time was creating a Claude project and dragging document assets into it (this was before Claude launched cowork). As inquiry complexity from customers increased, it became clear that agentic exploration of the codebase was critical, especially when sweeping through tags to track changes across releases. A lot of advisory work is helping customers understand the next most reasonable upgrade to consume to remediate a challenge or unlock a feature for integration. The agentic code searching has also allowed us to identify and confirm bugs with clear reproducers by working briefly with customers on their systems.

It’s a real privilege to work at a company where thinking about problems like these is encouraged, and where producing reference implementations to share is part of the culture. GitLab provides the Duo tokens that power my GitLab second brain and are allowing me to engage with customers and product with much greater efficacy.

My first attempt was summarization: pipe raw document text through Gemini CLI to populate a structured account template. The goal was to move context closer to daily inference by compressing each account’s history into something that fit in a context window. Knowingly lossy, but the tradeoff felt right: a 5-page account summary sitting in my digital garden series is more useful than 400 pages I have to scroll through. It helped with context switching, getting into the depth of a customer’s state quickly when moving across many accounts. But compressing 400 pages into 5 discards the specific details that make meeting notes valuable. “When did we first discuss pipeline execution policies?” requires the full history, not a summary of current state.

Why Not Just Give the LLM the File?

The alternative to vector search is feeding the entire document into an LLM’s context window. Both approaches use a model for synthesis, but they differ in how the relevant content reaches it.

Giving the LLM the full file works for a single document, but it scales poorly. Token cost grows linearly with document size, every query re-reads the entire corpus, and at 841 KB you’re already consuming a significant context window before the model starts reasoning. Vector search pre-indexes once, then each query retrieves only the relevant chunks. The tradeoff is that retrieval can miss things (as the failure modes below show), but the approach is composable: you can add documents, tune retrieval, and keep per-query costs constant.

We’re all doing knowledge work, and chunking/indexing meeting notes can feel like producing a derivative product. The goal isn’t to create a separate artifact. My GitLab digital garden series is a system for embedding my agency and workflow preferences into how I work, shared with peers. Work product consumed into that system should result in customer responses, issues, and merge requests back to the product itself. The pipeline is a means to that end, not the end.

A local system matters to me beyond just data sovereignty. My workflow pulls context locally to support that data model: stringing together several nearly headless utilities and enabling agent-based workflows via the AI harness. I don’t have API access to systems like Glean, and leaning on enterprise RAG for this kind of retrieval means token cost I can’t easily measure or control, with no way to integrate the results into local toolchains. A local pipeline I control is composable in ways a chat interface isn’t.

I started building before my organization rolled out Glean. Once Glean became available, it created an opportunity: a commercial enterprise search system indexing the same (and likely much larger) corpus. A meaningful benchmark was suddenly possible.

Architecture

The pipeline indexes one account’s notes (~10,800 lines, ~841 KB, 2020-2026) into LanceDB using BAAI/bge-base-en-v1.5 embeddings (768 dimensions, runs offline on Apple Silicon). Chunking splits on meeting date headers first, then paragraph boundaries. Each chunk carries a section_date metadata field. At query time, sentence-transformers embeds the question, LanceDB finds the nearest chunks, and GitLab Duo (DAP/Anthropic) synthesizes an answer.

The Harness Explorer project uses a similar embedding approach (all-MiniLM-L6-v2 via ONNX) for semantic similarity in a different domain. The pattern of “embed, compare, surface the nearest matches” turns out to be broadly useful.

Parameter Sweep

Before comparing against external systems, I ran a 36-configuration parameter sweep across four variables: embedding model, chunk size, chunk overlap, and top-k retrieval count. Each configuration re-indexed the corpus, retrieved chunks for 15 test questions across 5 categories, synthesized answers with Duo, and scored them using LLM-as-judge on accuracy, completeness, specificity, and citation quality.

The 5 question categories are designed to probe different retrieval failure modes:

Category	Tests	Example
Factual	Can the system retrieve known facts from meeting context?	“What infrastructure platform does this customer run on?”
People	Can it associate names with roles, decisions, and actions?	“Who is the primary infrastructure lead on this account?”
Temporal	Can it locate information within a specific time range?	“What was discussed in the most recent cadence call?”
Synthesis	Can it reason across multiple meetings to identify trends?	“How has this customer’s adoption maturity changed over time?”
Needle	Can it find a specific identifier buried in the corpus?	“What was the support ticket number for the outage last quarter?”

Parameter	Best	Worst	Impact
top-k	10	3	+19% overall
Chunk size	750 chars	3000 chars	Smaller chunks win
Overlap	0	200	No benefit with structure-aware splitting
Model	bge-base-en-v1.5	bge-small-en-v1.5	Marginal (~2%)

Winning configuration: bge-base-en-v1.5 / 750 chars / 0 overlap / top-10. The worst configuration (bge-small / 1500 chars / 200 overlap / top-3) happened to be my original defaults. Dead last out of 36.

Top-k dominates because more chunks means more chances to include the right information. Overlap doesn’t help because the chunking already splits on meeting date headers (no information lost at boundaries). Smaller chunks win because a 3000-character chunk covering three meeting topics produces a diluted embedding that represents the average of all three, weakening the signal for any one.

Benchmarking Against Enterprise Systems

The point of having a benchmark harness isn’t just to tune your own system. It’s to answer the question: how does this compare to what’s already available? I took the same 15 test questions from the parameter sweep, ran them through Gemini (native access to the full source document) and Glean (indexes documents, chat, CRM, and more), and scored all three systems with the same LLM-as-judge rubric.

System	Accuracy	Completeness	Specificity	Citations	Overall
Glean	4.00	4.40	4.67	3.53	4.15
Local RAG	2.33	2.33	3.27	3.47	2.85
Gemini	2.07	2.00	3.13	2.13	2.33

Glean scored 4.15, not 5.0. It’s a representative commercial baseline (the bar users experience daily), not a theoretical ceiling. The question is how close a local, offline, single-document system can get.

Category	Glean	Local	Gemini	Gap
Factual (3 Qs)	3.42	3.00	3.08	+0.42
People (3 Qs)	4.33	3.50	1.83	+0.83
Temporal (3 Qs)	4.58	2.33	3.50	+2.25
Synthesis (3 Qs)	3.42	2.75	2.00	+0.67
Needle (3 Qs)	5.00	2.67	1.25	+2.33

Two categories account for 70% of the total gap: temporal and needle queries. Glean scores a perfect 5.0 on needle queries (finding exact ticket numbers every time). On temporal queries, Glean understands “February 2026 cadence call” without effort.

Gemini scores below local RAG overall despite having the full 841 KB document as context. Its strength is temporal queries (3.50 vs local’s 2.33), likely because it can traverse the full document rather than relying on vector similarity. Its weakness is needle queries (1.25), suggesting that even full-document context doesn’t help when the model has to find a specific string in 841 KB of text.

Three Failure Modes

Temporal Blindness

Vector similarity has no concept of time. “February 2026 cadence call” retrieves 2020-era meetings because the embedding space maps “cadence call” to the same neighborhood regardless of date. I ranked all 503 chunks by L2 distance to the query “What was discussed in the February 2026 cadence call?” The correct chunk ranked 79th:

Vector search results:
  [1]  L2=0.39  "2020-05-06 Cadence Call"           <- wrong year
  [2]  L2=0.42  "2020-04-17 Cadence Call"           <- wrong year
  ...
  [79] L2=0.75  "Feb 4, 2026 | Cadence"             <- correct

Keyword/Entity Blindness

Dense retrieval cannot find specific strings. Ticket numbers, issue references, and configuration values are opaque to embeddings. “Zendesk #9786” and “issue #458832” appear in exactly one chunk each, but vector similarity between the query and those chunks is low because the embedding captures topic similarity, not string containment.

Context Window Starvation

Even when retrieval finds the right meeting, it may retrieve the header chunk (attendees, agenda) but miss the sibling chunk (discussion notes, action items). The Feb 4 cadence header is 364 characters. The notes are 4,610 characters, in a separate chunk. With top-10 retrieval at 750-char chunks, the synthesis model sees the meeting existed but not what was discussed.

Meeting Section	Size	Without Expansion	With Expansion
Header (attendees, agenda)	364 chars	Retrieved	Retrieved
Discussion notes	4,610 chars	Missed	Retrieved
Action items	~800 chars	Missed	Retrieved

Closing the Gap

Each failure mode has a targeted fix. None require changing the embedding model or retraining anything.

Date pre-filtering. For temporal queries, extract date ranges from the query using regex (“February 2026” becomes section_date BETWEEN '2026-02-01' AND '2026-02-28'), then apply as a SQL pre-filter on LanceDB before vector search. The correct chunk moves from rank 79 to rank 1. Zero LLM overhead.

Hybrid search. For needle queries, combine vector similarity with Tantivy-based full-text search using reciprocal rank fusion. Full-text search trivially finds “#9786” by string match. LanceDB supports this natively:

table.search(query, query_type="hybrid")
     .limit(10)
     .rerank(RRFReranker())

Parent-chunk expansion. When a chunk is retrieved, also pull all sibling chunks from the same meeting section, identified by matching (account, section_date, doc_name) metadata. If the header chunk is retrieved, the notes and action items come along automatically.

Combined Results

Category	Glean	After Fixes	Before Fixes	Delta
Factual	3.42	2.92	3.00	-0.08
People	4.33	3.75	3.50	+0.25
Temporal	4.58	3.17	2.33	+0.84
Synthesis	3.42	3.83	2.75	+1.08
Needle	5.00	3.17	2.67	+0.50
Overall	4.15	3.37	2.85	+0.52

The three fixes close 40% of the gap to Glean (2.85 to 3.37). The biggest gains came from exactly where the fixes targeted: synthesis +1.08 (full meeting context instead of fragments), temporal +0.84 (rank 79 eliminated), needle +0.50 (exact-match chunks surfaced). The remaining gap (0.78 points) concentrates in temporal (1.41 remaining) and needle (1.83 remaining).

The Overengineering Trap

With 40% of the gap closed, I tried three more retrieval techniques from the literature: cross-encoder reranking (retrieve 50 candidates with the bi-encoder, rerank to 10 with cross-encoder/ms-marco-MiniLM-L-6-v2), soft temporal scoring (multiply retrieval scores by a date-proximity bonus), and query expansion (extract identifiers via regex, run targeted FTS-only searches, merge results).

Variant	Features Added	Score	Delta
Phase 1 (baseline)	hybrid + date + expand	3.37	-
+ Rerank only	cross-encoder ms-marco	3.18	-0.19
+ Temporal-boost only	date-proximity scoring	3.22	-0.15
+ Query-expand only	identifier extraction + FTS	3.27	-0.10
+ TB + QE	temporal + query-expand	3.28	-0.09
+ All three	rerank + TB + QE	3.23	-0.14

Every Phase 2 feature made things worse. The cross-encoder (trained on web search, not meeting notes) confidently promotes topically relevant passages from the wrong meeting and demotes date-filtered results the bi-encoder correctly surfaced. The temporal boost over-weights date proximity versus semantic relevance when the date filter already surfaces the right meeting. The query expansion regex picks up tokens that aren’t identifiers in context, adding noise to the FTS merge.

The distinction matters: Phase 1 worked because each fix addressed a specific, diagnosed failure mode with hard evidence (rank 79 to rank 1, zero hybrid results to exact match, missing siblings to full meeting). Phase 2 was plausible heuristics applied without the same diagnostic grounding. Each additional retrieval transform introduces error risk. This maps onto something the merge request as friction post explores: the difference between intervention with evidence and intervention by instinct.

Interrogating Glean

Before investing in more retrieval features, I designed probe questions to test specific hypotheses about Glean’s retrieval architecture.

Probe: “What data sources do you have about this customer?” Glean listed 20+ sources across 8 platforms: meeting notes, shared documents, cloud storage PDFs, issue trackers, internal wiki pages, CRM records, email threads, calendar events, and presentation decks. I’d been benchmarking a single-document RAG against a system indexing 20+ sources across 8 platforms. The comparison was never apples-to-apples.

Local RAG: 1 source

• Meeting notes document

Glean: 20+ sources, 8 platforms

• 7 shared documents
• 3 cloud storage PDFs
• 2 issue tracker items
• 3 internal wiki pages
• 2 CRM records
• 3+ email threads
• 4+ calendar events
• 1 presentation deck

Probe: “Show me the passages you’re using for questions about a specific topic.” Glean quoted 10 verbatim passages from 10 different meeting sections spanning 18 months, each 100-300 words, organized chronologically. This revealed that Glean retrieves full meeting sections (not fixed-size chunks) and many of them (at least 10 sections for a broad topic query). That’s roughly 30-50 KB of context fed to the synthesis model, versus my 7.5 KB.

Glean’s advantage breaks down to three factors: multi-source corpus (~40% of gap), larger context windows (~40%), and section-level parsing (~20%). Going into this I expected Glean was doing something architecturally deeper: pre-building structured meeting records, maintaining an entity graph, or running multi-pass retrieval. What it does not appear to do is any of that. Its answers cite verbatim passages, not structured fields. As far as I can tell, the advantage is effectively brute-force token use: index more sources, retrieve more chunks, feed a larger context window to the synthesis model. That was surprising. It also reinforces why avoiding that token cost matters if you can get close enough with a local system.

What Remains

Retrieval is the whole game. The synthesis model matters far less than what chunks reach it. Embedding model choice made a 2% difference. Date filtering alone moved a critical chunk from rank 79 to rank 1.

Build the benchmark before optimizing. Without hard numbers, I would have spent time on embedding model selection or prompt tuning. The harness pointed directly at the three failure modes that matter and told me exactly which questions to fix. The harness itself is the real product.

Diagnose before you build. Phase 1 fixes worked because each one targeted a specific, measured failure. Phase 2 was me getting excited about optimization techniques without first measuring whether they addressed real problems. That was a waste.

The unfair comparison is the interesting one. Benchmarking a single-document local system against enterprise search that indexes documents, email, calendar, CRM, and issue trackers sounds unfair. It is. But the question is whether 81% of Glean’s score running fully offline is close enough to avoid the per-query token cost of routing every retrieval question through an enterprise system I can’t integrate into my local toolchain.

The things I want to build list has adjacent ideas around this.