AI search
Retrieval-augmented generation for the working schlub
2024-02-06 — Invalid Date
In Which a Personal Repository Is Queried Through Both Classic Term-Frequency Statistics and Modern Neural Embeddings, Demonstrating That Local Search Efficacy Is Achieved Without a Dedicated Database.
This page collects notes on AI search over a bounded corpus (i.e., something small like this blog, not big like the internet). Writing a general theory of information retrieval for the information age is out of scope for my notebook. What I do here is write down some worked-example pipelines that were useful for me.
The first set of tricks is a whole-document similarity index (“suspiciously similar posts” on this blog).
The second is a chunked retrieval search (“find where I wrote about X”). Both rely on vector embeddings, but they make different architectural choices.
The underlying problem is the same one that classical information retrieval solves—finding documents similar to a query, but with neural network fairy dust sprinkled on top.
The classic tools (Lucene, Xapian, Sphinx) hand-crafted an embedding vector space from term frequencies (back in my day it was TF-IDF—term frequency–inverse document frequency). In the new tools, learned neural embeddings replace those hand-crafted ones. The geometry is the same (cosine similarity over high-dimensional vectors), but the learned vectors capture semantic similarity more directly than lexical overlap did, which leads to a dramatic improvement in performance.
Traveller beware: Most of the text below has been translated from my dev notes by an LLM (i.e. I did it, and then got the LLM to write it up) and I have not reviewed it. Bug reports welcome.
1 Similarity searching this blog
Those “suspiciously similar posts” links at the top of the page use a kernel similarity search, an off-the-shelf embedding, and some numpy.
1.1 How it works
Embedding models! We embed each post as a single vector (1024 dimensions via mxbai-embed-large, run in-process through sentence-transformers). The embedding model sees only the first 512 tokens of each post — roughly the title, categories, and a paragraph or two. That’s the entire “understanding” of the post, which seems, surprisingly, sufficient for finding related content quite well.
The similarity computation is a dot product of L2-normalized vectors, which gives cosine similarity:
Q is the query matrix (the posts we want neighbours for), E is the full document embedding matrix, and NumPy handles the rest. For ~1700 posts with 1024-dimensional embeddings, the matrix is about 7 MB. I mildly optimized top-k selection via np.argpartition, which is \(\mathcal{O}(N)\) per query rather than \(\mathcal{O}(N \log N)\) for a full sort—but at this scale, even a full sort would be fast.
The embeddings are stored compressed in an .npz file (as float16 on disk). I also stash per-document metadata (title, content hash, categories). Any time I want to compute similarity, I need to load the entire thing into memory, so it’s worth performing queries in batch. Incremental updates are cheap because I store a blake2s hash of the truncated text, and we skip unchanged posts during re-indexing.
When I publish the blog, I output one JSON file per page in related/, which the client-side template fetches to render the “suspiciously similar” links.
1.2 Choosing embedding model
At the time of implementation, there were two top-ranked open models: nomic-ai/nomic-embed-text-v1.5 (8192 token context) and mixedbread-ai/mxbai-embed-large-v1 (512 token context). I auditioned both by seeing if the “similar posts” they estimated “felt right”. Counter-intuitively, the mxbai felt way more natural, despite seeing far less of each post. I suspect this is because for similarity (as opposed to keyword search), the title and opening paragraph carry most of the topical signal, and a shorter context forces the model to focus on that signal rather than diluting it with whatever weird tangent I have gone on in the body text.
I’m curious about the SPECTER2 embeddings, which apparently produce good embeddings for science papers, but the API is rather different, so I didn’t hot-swap it in for testing.
1.3 At teensy blog scale
The entire approach—embed everything, dump to a flat NumPy array, brute-force cosine distance—is viable because ~2000 documents are tiny by modern standards. There’s no approximate nearest-neighbour index, no vector database, and no sharding. The full pairwise similarity matrix is 2000×2000, which fits in L2 cache.
The script is open source. You can download it from similar_posts_static_site.py.
However the version that runs this site has many improvements I did not include there, sorry. Nag me if you want the latest version.
2 Local semantic search with QMD
The similarity index described above is great for finding “what posts are related to this one?” but it doesn’t find stuff by topic. The difference is that the former is a similarity problem, whereas here we want to solve a relevance problem.
For example, similarity works best with embeddings of whole documents (as I do with mxbai-embed-large), while retrieval benefits from sub-document chunking so we can find the right passage inside a long post.
A classic trick that is surprisingly still in use alongside these vector embeddings is BM25 (“Best Matching 25”). BM25 is a term-frequency scoring function from the 1990s that improves on raw TF-IDF by adding document length normalization and diminishing returns for repeated terms. It’s the default ranking algorithm in Lucene, Elasticsearch, and most traditional search engines. BM25 is fast and surprisingly effective for keyword queries — if someone searches for “particle filter” and a document contains those exact words, BM25 will find it reliably. It is less good at semantic similarity: it cannot know that “sequential Monte Carlo” means the same thing, because it only counts words, it doesn’t understand them. BM25 remains useful as a complementary signal alongside the neural embedding to inform search.
QMD (by Tobi Lütke) is a local CLI search engine that combines BM25, vector semantic search, and LLM reranking. It has its own opinions about embedding models. Rather than reusing the mxbai-embed-large I use for similarity, it defaults to its own blessed set: embeddinggemma-300M to embed, Qwen3-Reranker-0.6B to rerank, and a fine-tuned 1.7B model to expand queries. The models are relatively compact: 2.1 GB of GGUF weights sitting in ~/.cache/qmd/models/, which duplicates capability I already have installed elsewhere but is not the worst. QMD_EMBED_MODEL overrides the embedding model; the real value add for that would be if the corpus is not mostly English, since embeddinggemma is English-optimized and Qwen3-Embedding covers rather more languages. Embeddings are not comparable across models, so switching models means re-embedding the corpus with qmd embed -f.
2.1 Installation
I ran my own fork for a while, carrying a few bugfixes that upstream had not merged. Upstream merged them in August 2026, so now I track its main branch.
The published npm release is still 2.5.3, from May 2026. It mangles underscores in every path it prints, so a hit in notebook/particle_filters.qmd gets reported as notebook/particle-filters.qmd.
2.2 Indexing this blog
I register the blog content directories as a collection and generate embeddings. QMD defaults to **/*.md, so without --mask it will only find plain markdown files. For this blog, I declared --mask "**/*.qmd" explicitly so we organize the Quarto content, which makes up 90% of everything.
By default, QMD keeps one index for the whole machine, in ~/.cache/qmd/index.sqlite, configured from ~/.config/qmd/index.yml. qmd init instead puts both inside the project, at .qmd/. I prefer that local configuration, since I can track and version the configuration with the content. This also works with git worktrees. The index itself is a rebuildable cache, so it stays out of version control.
Configuring the CLI via the config YAML file is less error-prone than doing so via CLI flags. Here is mine
ignore seems to be YAML-only — I cannot find a CLI to set it. Here I’m ignoring the digests: they’re machine-written summaries of the site, which shouldn’t typically be indexed alongside the content they summarize.
context attaches a human-written description to a collection or a path prefix. We prepend it to every result so the model ranking them knows what kind of corpus it’s looking at.
The embed step runs a local GGUF embedding model via llama.cpp. It chunks documents at paragraph boundaries by default, which seems fine. --chunk-strategy auto switches to tree-sitter chunking at function and class boundaries for various programming languages.
2.3 Search
QMD exposes three search types:
The hybrid query mode is usually what I want.
Each hit prints a path, a short docid like #bb2e21, a title, and a snippet. qmd get "#bb2e21" returns a document, and qmd multi-get "#a1b2c3, #d4e5f6" returns several at once. Other useful flags: -n for the number of results (default 5, or 20 in the machine-readable formats), --format to choose between cli, json, md, csv, xml, and a bare files list, --full to dump whole documents rather than snippets, --min-score with --all to take everything above a threshold, and --no-rerank to skip the LLM pass for worse, quicker results.
Given a plain string, query asks a fine-tuned 1.7B model to expand it into alternative phrasings before searching. I can skip that and write the expansion myself:
intent: is context for the reranker rather than a search in its own right; lex: goes to BM25, vec: to the embedding, and hyde: is a sentence resembling the answer I hope exists, which is then embedded and matched against the corpus. This is also the mode we use from an agent, which presumably knows the domain vocabulary and the nearby-but-wrong concepts better than the tiny 1.7B expander guessing from four words.
These all improve search results. lex: QMD collection mask returned a post about fashion at 88% confidence, because (I presume) clothes come in collections and masks are also a garment. BM25 does not know which sense I meant, and the reranker was happy enough with a document that contained both words.
2.4 Opening results in my editor
QMD emits clickable OSC 8 terminal hyperlinks in its search results. In a modern terminal (iTerm2, Kitty, WezTerm, Ghostty), each result is a clickable link that opens the file at the matching line in my editor — like <a href> in HTML, but for the terminal. The URI scheme is configurable through QMD_EDITOR_URI or an editor_uri line in the config, so it works with VS Code (vscode://file/), Cursor, Zed, and Sublime.
NB: that doesn’t work on the published 2.5.3 version. Install from main instead. For anyone stuck on the release, the docid offers a reliable fallback — it appears with every hit, and qmd get "#bb2e21" resolves regardless of what the path says.
2.5 Driving it from an agent
QMD runs as an MCP server (qmd mcp, or qmd mcp --http --daemon for a persistent one that keeps the models resident), exposing query, get, multi_get, and status as tools. Registering it in Claude Code is one line, claude mcp add qmd -- qmd mcp, or a .mcp.json in the repo root.
I deregistered it again. The CLI provides the same search but is more compact and powerful. An agent that runs qmd query in a shell gets the flags the MCP tools don’t expose. The MCP layer might be worth it if the client can’t run shell commands. qmd skill install drops a version-matched instruction file into .agents/skills/.
2.6 Alternative: txtai for composable pipelines
QMD is opinionated — it bundles its own embedding model, chunker, BM25 index, and reranker into one tool.
txtai takes the opposite approach: it’s a Python framework where each stage of the pipeline — chunking, embedding, indexing, retrieval, reranking — is a swappable component. We could plug in mxbai-embed-large (reusing the same model as the similarity pipeline), write a custom chunker that understands .qmd frontmatter and fenced code blocks, and choose our own ANN backend. It also exposes a REST API, so wrapping it as an MCP server would be straightforward.
The tradeoff is assembly time: QMD took five minutes 3 hours to set up; a txtai pipeline would take at least an extra afternoon. Having the option to swap in something that understands local document format weirdness might be nice.
As of early 2026, there is AFAICT no good VS Code extension for semantic search over prose markdown. The extensions that exist — Zilliz Semantic Code Search, sturdy-dev/semantic-code-search — are designed around code. They list Markdown as a supported filetype, but they don’t chunk prose at paragraph boundaries or handle frontmatter, math blocks, or callouts, i.e. they do not do human speaky words.
The markdown knowledge base extensions (Foam, Markdown Memo) handle wikilinks and graph visualization but don’t do vector search at all.
3 Really big corpora
Both examples above work because the corpus is small — \(<10^5\) documents/chunks. At that scale, the choice of infrastructure barely matters; everything fits in memory, and every query is fast. That changes as we scale up.
A vector search system has three stages, as we have already seen:
- Chunking — split documents into passages.
- Embedding — map each chunk to a vector.
- Indexing and retrieval — given a query vector, find the nearest chunks.
Retrieval-augmented generation (RAG) adds a fourth stage: feed the retrieved chunks to an LLM as context and generate a synthesized answer.
At the small end (this blog), the “stack” is:
sentence-transformers→ numpy.npz→Q @ E.T→ JSON files
: use pgvector for retrieval
t: your team already runs Postgres. Adding Elasticsearch means new ops burden, new failure modes, new on-call knowledge. pgvector’s recall is within 3% of Elasticsearch on our corpus size (benchmarked last month). the 3% recall difference costs less than the ops burden
a1: use Elasticsearch — better recall on paper, wrong tradeoff for this team’s actual constraints (alternative)
a2: what if corpus grows 10x? — does the recall gap widen? (stress test)
a3: pgvector’s recall is within 3%, based on what benchmark? — is the benchmark representative of production query patterns? (assumption-surfacing)
a4: skip vector search, use hybrid keyword + rerank — sidesteps the infra question entirely (alternative)
Now here’s the thing about generating a2, a3, a4: a language model is legitimately useful for this. Not for a1 (the recommendation) — you (or someone who knows the domain) should own that call. But the challenge set benefits from a kind of adversarial creativity that’s expensive for the person deep in the decision to generate for themselves, precisely because they’re deep in the decision. This is the same reason human red teams work: the person who built the system is bad at attacking it, not from lack of skill, but from lack of distance.
I’ve been building a tool to test this. You write the claim and rationale, it generates challenges, and critically, it scores them before you see them — for redundancy against each other, for relevance to the actual claim, for whether they attack the decision-relevant part of the reasoning or something peripheral. Bad challenges get filtered before they waste your time. What survives is maybe 2-3 challenges instead of 10, and they’re the ones that would actually change the recommendation if they landed.
Embedding API → Vector database (Pinecone, Qdrant, Weaviate) → ANN retrieval → Reranker → LLM
The Algolia search that powers the search box at the top of the page presumably uses similar technology under the hood. However, a third party runs it and serves the content from their own servers, so I can’t really speak to what they’re doing.
4 Interesting embeddings
4.1 Generic text embeddings
- Nomic embeddings are small, fast, and open. I trialled them for this blog and they were decent but not amazing, even though they have a large context window.
- Mixed bread/
mxbaiembeddings are generated by a relatively large model that uses a small number of tokens. Counter-intuitively, they worked great for classifying the text of this blog, even though they only look at the first 512 tokens.
4.2 Specialized for scientific text
SPECTER2: Adapting scientific document embeddings to multiple fields and task formats:
Models like SPECTER and SciNCL are adept at embedding scientific documents as they are specifically trained so that papers close to each other in the citation network are close in the embedding space as well. For each of these models, the input paper text is represented by a combination of its title and abstract. SPECTER, released in 2020, supplies embeddings for a variety of our offerings at Semantic Scholar - user research feeds, author name disambiguation, paper clustering, and many more! Along with SPECTER, we also released SciDocs - a benchmark of 7 tasks for evaluating the efficacy of scientific document embeddings. SciNCL, which came out last year, improved upon SPECTER by relying on nearest-neighbour sampling rather than hard citation links to generate training examples.
This model and its ilk are truly targeted at research discovery, and they are so good at it that we might argue they have “solved” the knowledge topology problem for scientific papers.
I implemented a search engine for ICLR 2025 using the SPECTER2 embeddings and was impressed with the quality of the results. Note that the API differs slightly from the default huggingface API used by mxbai et al.; we need to use the “adapters” library.
5 Tools
ChromaDB is a vector database with a focus on search and retrieval. I used it to store vector embeddings to note “similar posts” on this site and I can report that it was incredibly simple for my use case, and that it scaled well to thousands of documents, at least. I actually replaced it with something even simpler. It’s based on sqlite.
6 Searching the internet rather than a corpus
Everything above is about a corpus we host ourselves. For search engines pointed at the internet at large, there are two other pages: internet search covers those aimed at humans, and web search for machines covers the scrape-and-retrieve APIs (Jina, Firecrawl, Tavily and friends) that agents call.