AI search
Retrieval-augmented generation for the working schlub
2024-02-06 — 2026-07-24
Wherein the Author Compares Whole-Document Embedding Against Chunked Retrieval for Small Corpora, and Notes That BM25 Confuses “Masks” of Cloth With Masks of Search Queries.
This page collects notes on AI search over a bounded corpus (i.e. something small like this blog, not big like the internet).
The first set of tricks is a whole-document similarity index (“suspiciously similar posts” on this blog).
The second is a chunked retrieval search (“find where I wrote about X”). Both rely on vector embeddings but they make different architectural choices.
The underlying problem is the same one that classical information retrieval solves — find documents similar to a query, but with neural network fairy dust sprinkled on top.
The old tools (Lucene, Xapian, Sphinx) hand-crafted an embedding vector space from term frequencies (back in my day it was TF-IDF — term frequency–inverse document frequency). In the new tools learned neural embeddings replace those hand-crafted ones. The geometry is the same (cosine similarity over high-dimensional vectors), but the learned ones capture semantic similarity rather than lexical overlap, which leads to a dramatic improvement in performance.
BM25 (“Best Matching 25”) is a term-frequency scoring function from the 1990s that improves on raw TF-IDF by adding document length normalization and diminishing returns for repeated terms. It’s the default ranking algorithm in Lucene, Elasticsearch, and most traditional search engines. BM25 is fast and surprisingly hard to beat for keyword queries — if someone searches for “particle filter” and a document contains those exact words, BM25 will find it reliably. It is less good at semantic similarity: it can’t know that “sequential Monte Carlo” means the same thing, because it only counts words, it doesn’t understand them. BM25 is still useful as a complementary signal to the neural embedding to improve search sort order, so tools like QMD combine both.
For background on vector databases and embeddings see those respective pages.
Traveller beware: Most of the text below has been translated from my dev notes by an LLM (i.e. I did it, and then got the LLM to write it up) and I have not reviewed it. Bug reports welcome.
1 Similarity searching this blog
Those “suspiciously similar posts” links at the top of the page are generated by a kernel similarity search, an off-the-shelf embedding and some numpy.
1.1 How it works
Each post is embedded as a single vector (1024 dimensions via mxbai-embed-large, run in-process through sentence-transformers — no server, no daemon; the model downloads to the HuggingFace cache on first run). The embedding model sees only the first 512 tokens of each post — roughly the title, categories, and a paragraph or two. That’s the entire “understanding” of the post, which seems, surprisingly, sufficient for finding related content pretty well.
The similarity computation is a dot product of L2-normalised vectors, which gives cosine similarity:
That’s it. Q is the query matrix (the posts we want neighbours for), E is the full document embedding matrix, and numpy handles the rest. For ~1700 posts with 1024-dimensional embeddings the matrix is about 7 MB. I mildly optimized Top-k selection via np.argpartition, which is \(\mathcal{O}(N)\) per query rather than \(\mathcal{O}(N \log N)\) for a full sort — but at this scale even a full sort would be fast.
The embeddings are stored compressed in an .npz file (as float16 on disk). I also stash per-document metadata (title, content hash, categories). I load the entire thing up any time I want to compute similarity, so it’s worth doing queries in batch. Incremental updates are cheap because I store a blake2s hash of the truncated text, and unchanged posts are skipped on re-index.
When I publish the blog, I output one JSON file per page in related/, which the client-side template fetches to render the “suspiciously similar” links.
1.2 Embedding model
I auditioned two models: nomic-ai/nomic-embed-text-v1.5 (8192 token context) and mixedbread-ai/mxbai-embed-large-v1 (512 token context). Counter-intuitively, the mxbai “felt” way more natural, despite seeing far less of each post. I suspect this is because for similarity (as opposed to keyword search), the title and opening paragraph carry most of the topical signal, and a shorter context forces the model to focus on that signal rather than diluting it with the body text.
I’m curious about the SPECTER2 embeddings, which apparently produce good embeddings for science papers, but the API is rather different so I didn’t hot-swap it in for testing.
1.3 At teensy blog scale
The entire approach — embed everything, dump to a flat numpy array, brute-force cosine distance — is viable because ~2000 documents is tiny. There’s no approximate nearest-neighbour index, no vector database, no sharding. The full pairwise similarity matrix is 2000×2000, which fits in L2 cache.
The script is open source. You can download it from similar_posts_static_site.py.
However the version that runs this site has many improvements I did not include there, sorry. Nag me if you want the latest version.
2 Local semantic search with QMD
The similarity index described above is great for finding “what posts are related to this one?” but it doesn’t find stuff by topic, which is not a similarity problem but a relevance problem.
For example, similarity works best with embeddings of the whole document (as I do with mxbai-embed-large), while retrieval benefits from sub-document chunking so we can find the right passage inside a long post.
QMD (by Tobi Lütke) is a local CLI search engine that combines BM25 full-text search, vector semantic search, and LLM reranking. It has its own opinions about embedding models rather than reusing the mxbai-embed-large I use for similarity: embeddinggemma-300M to embed, Qwen3-Reranker-0.6B to rerank, and a fine-tuned 1.7B model to expand queries. This might be correct: different models for different tasks. The cost is 2.1 GB of GGUF weights sitting in ~/.cache/qmd/models/, which duplicates capability I already have installed elsewhere. QMD_EMBED_MODEL overrides the embedding model — worth knowing if a corpus is not mostly English, since embeddinggemma is English-optimized and Qwen3-Embedding covers rather more languages. Vectors are not comparable across models, so switching one means re-embedding the corpus with qmd embed -f.
2.1 Installation
Or run without installing via npx @tobilu/qmd.
2.2 Indexing this blog
I register the blog content directories as a collection and generate embeddings. QMD defaults to **/*.md, so without --mask it will only find plain markdown files. For this blog, we declare --mask “**/*.qmd” explicitly so we pick up the Quarto content. Watch out: the command silently ignores unknown flags.
By default, QMD keeps one index for the whole machine, in ~/.cache/qmd/index.sqlite, configured from ~/.config/qmd/index.yml. qmd init instead puts both inside the project, at .qmd/. I prefer that here: the config becomes a tracked file that travels with the repo, and each git worktree gets its own index rather than quietly answering with the contents of the main checkout. The index itself is a rebuildable cache, so it stays out of version control.
Everything the CLI knows lives in that one YAML file, and editing it directly is easier than remembering which subcommand sets what:
ignore is the interesting key, and it is YAML-only — no subcommand sets it. Excluding the digests matters more than it might sound: they are machine-written summaries of what I changed, so a search for a topic returns a paraphrase of the post I actually wanted, ranked above the post itself.
context attaches a human-written description to a collection or a path prefix, and it rides along with every result so the model ranking them knows what kind of corpus it is looking at. The upstream project considers this the important feature. Keep it short, because it prints in full with every single hit.
The embed step runs a local GGUF embedding model via llama.cpp. It chunks documents at paragraph boundaries by default, which is the right granularity for prose blog posts. --chunk-strategy auto switches to tree-sitter chunking at function and class boundaries for TypeScript, JavaScript, Python, Go and Rust; prose stays on the regex chunker regardless.
2.3 Search
QMD exposes three search tiers:
The hybrid query mode is usually what I want.
Each hit prints a path, a short docid like #bb2e21, a title and a snippet. qmd get “#bb2e21” returns the document, and qmd multi-get “#a1b2c3, #d4e5f6” returns several at once. Other useful flags: -n for the number of results (default 5, or 20 in the machine-readable formats), --format to choose between cli, json, md, csv, xml and a bare files list, --full to dump whole documents rather than snippets, --min-score with --all to take everything above a threshold, and --no-rerank to skip the LLM pass for worse, quicker results.
Given a plain string, query asks a fine-tuned 1.7B model to expand it into alternative phrasings before searching. I can skip that and write the expansion myself, which is what I do when I know the vocabulary:
intent: is context for the reranker rather than a search in its own right; lex: goes to BM25, vec: to the embedding, and hyde: is a sentence resembling the answer we hope exists, which then gets embedded and matched against the corpus. The first line gets double weight. This is also the mode to use from an agent, which knows the domain vocabulary and the nearby-but-wrong concepts far better than a 1.7B expander guessing from four words.
Writing the lexical line myself is an art: lex: QMD collection mask returned a post about fashion at 88% confidence, because (I presume) clothes come in collections and masks are also a garment. BM25 does not know which sense I meant, and the reranker was happy enough with a document that contained both words. Hand-written keywords need to be the rare ones, not the ambiguous ones.
2.4 Opening results in my editor
QMD emits clickable OSC 8 terminal hyperlinks in its search results. Which is to say, in a modern terminal (iTerm2, Kitty, WezTerm, Ghostty), each result is a clickable link that opens the file at the matching line in my editor — like <a href> in HTML, but in the terminal. The URI scheme is configurable through QMD_EDITOR_URI or an editor_uri line in the config, so it works with VS Code (vscode://file/), Cursor, Zed and Sublime. When output is piped rather than displayed, the escape sequences are dropped and we get plain text.
At least, on a recent version: none of that is on npm yet. The released version was still 2.5.3 last I checked, so a plain npm install -g @tobilu/qmd gets slightly broken old behaviour, and the fix only arrives by building from source. For anyone stuck on the release, the docid is the reliable handle — it appears with every hit and qmd get “#bb2e21” resolves regardless of what the path says.
2.5 Driving it from an agent
QMD runs as an MCP server (qmd mcp, or qmd mcp --http --daemon for a persistent one that keeps the models resident), exposing query, get, multi_get and status as tools. Registering it in Claude Code is one line, claude mcp add qmd -- qmd mcp, or a .mcp.json in the repo root.
I removed it again. The CLI is the same search, and an agent that runs qmd query in a shell gets the flags the MCP tools don’t expose, at the cost of nothing except a line in the permissions config so it doesn’t ask before every search. The MCP layer is worth it when the client can’t run shell commands, which is not the case here. qmd skill install drops a version-matched instruction file into .agents/skills/, for anyone who would rather the agent read upstream’s guidance than their own.
What actually made the difference was writing down the two things above that are not guessable: that display paths are slugified, and that the structured intent:/lex:/vec:/hyde: form exists. An agent left to itself pastes the user’s words into qmd query and then tries to open a filename that isn’t there.
2.6 Alternative: txtai for composable pipelines
QMD is opinionated — it bundles its own embedding model, chunker, BM25 index, and reranker into one tool. That’s a strength when we want to get running quickly, but a limitation when its choices don’t suit the corpus.
txtai takes the opposite approach: it’s a Python framework where each stage of the pipeline — chunking, embedding, indexing, retrieval, reranking — is a swappable component. We could plug in mxbai-embed-large (reusing the same model as the similarity pipeline), write a custom chunker that understands .qmd frontmatter and fenced code blocks, and choose our own ANN backend. It also exposes a REST API, so wrapping it as an MCP server would be straightforward.
The tradeoff is assembly time: QMD took five minutes to set up; a txtai pipeline would take an afternoon. But if QMD’s regex paragraph chunker starts mangling math blocks or YAML frontmatter, having the option to swap in something that understands the local document format weirdness might be nice?
As of early 2026, there is AFAICT no good VS Code extension for semantic search over prose markdown. The extensions that exist — Zilliz Semantic Code Search, sturdy-dev/semantic-code-search — are designed around code. They list Markdown as a supported filetype, but they don’t chunk prose at paragraph boundaries or handle frontmatter, math blocks, or callouts, i.e. they do not do human speaky words.
The markdown knowledge base extensions (Foam, Markdown Memo) handle wikilinks and graph visualization but don’t do vector search at all.
3 Really big corpora
Both examples above work because the corpus is small — \(<10^5\) documents/chunks. At that scale, the choice of infrastructure barely matters; everything fits in memory, and every query is fast. That changes as we scale up.
3.1 Decomposing the retrieval pipeline
A vector search system has three stages, which we already saw:
- Chunking — split documents into passages.
- Embedding — map each chunk to a vector.
- Indexing and retrieval — given a query vector, find the nearest chunks.
Retrieval-augmented generation (RAG) adds a fourth stage: feed the retrieved chunks to an LLM as context and generate a synthesized answer.
3.2 Infrastructure spectrum
At the small end (this blog), the “stack” is:
sentence-transformers→ numpy.npz→Q @ E.T→ JSON files
At the large end (production RAG), we’d see something like:
Embedding API → Vector database (Pinecone, Qdrant, Weaviate) → ANN retrieval → Reranker → LLM
The Algolia search that powers the search box at the top of the page presumably uses similar technology under the hood. However, it’s run by a third party who serves the content from their own servers, so I can’t really speak to what they’re doing.
4 Interesting embeddings
4.1 Generic text embeddings
- Nomic embeddings: Introducing Nomic Embed: A Truly Open Embedding Model are small, fast, and open. I trialled them for this blog and they were OK but not amazing, even though they have a big context window.
- mxbai embeddings: Open Source Strikes Bread - New Fluffy Embedding Model | Mixedbread These embeddings are generated by a relatively large model that uses a relatively small number of tokens. Counter-intuitively, they were great at classifying the text of this blog, even though they only look at the first 512 tokens.
4.2 Specialized for scientific text
SPECTER2: Adapting scientific document embeddings to multiple fields and task formats:
Models like SPECTER and SciNCL are adept at embedding scientific documents as they are specifically trained so that papers close to each other in the citation network are close in the embedding space as well. For each of these models, the input paper text is represented by a combination of its title and abstract. SPECTER, released in 2020, supplies embeddings for a variety of our offerings at Semantic Scholar - user research feeds, author name disambiguation, paper clustering, and many more! Along with SPECTER, we also released SciDocs - a benchmark of 7 tasks for evaluating the efficacy of scientific document embeddings. SciNCL, which came out last year, improved upon SPECTER by relying on nearest-neighbour sampling rather than hard citation links to generate training examples.
This model and its ilk are truly targeted at research discovery, and they are so good at it that we might argue they have “solved” the knowledge topology problem for scientific papers.
I implemented a search engine for ICLR 2025 using the SPECTER2 embeddings and I was impressed with the quality of the results. Note that the API is a little different from the default huggingface API used by mxbai et al.; we need to use the “adapters” library.
5 Tools
ChromaDB is a vector database with a focus on search and retrieval. I used it to store vector embeddings to note “similar posts” on this site and I can report it was incredibly simple for my use case, and scaled well to thousands of documents, at least. I actually replaced it with something even simpler. It’s based on sqlite.
6 Searching the internet rather than a corpus
Everything above is about a corpus we host ourselves. For search engines pointed at the internet at large, there are two other pages: internet search covers the ones aimed at humans, and web search for machines covers the scrape-and-retrieve APIs (Jina, Firecrawl, Tavily and friends) that agents call.