Web search for machines
Search and scraping APIs for agents, which do not click links
2026-07-24 — 2026-11-12
Wherein Firecrawl Is Eschewed for the Cheaper Jina, Brave’s Index Is Queried for Five Dollars Per Thousand Units, and Kagi’s Personal Search Blocks Are Applied to Agentic Tasks Alongside Tavily’s Snippet Extraction.
On this page
An internet search engine for humans returns an HTML page of links, on the reasonable assumption that a human will click a few of them to read. An agent does not click. It would much rather not have to parse the HTML itself, and instead get text that’s already stripped of nonsense, chunked, and ideally carrying enough provenance that the model can say where a claim came from. There is a whole market of vendors selling that AI-facing product now.
For the retrieval machinery underneath — chunking, embeddings, BM25, reranking — see AI search.
1 Jina
Below I mention Firecrawl as a web scraping tool. It is kind of the default, mostly because it is famous.
However, it is too gold-plated for me, so I have bumped it in favour of Jina. As far as I can tell, Firecrawl’s pricing reflects a premium for using browsers for scraping, which is a good way to get clean content from dynamic, JS-first websites, but it’s also slow and resource-intensive, and overkill for 80% of my use cases.
Jina doesn’t do that but it is way cheaper and 80/20s my use cases.
2 Firecrawl
Firecrawl is a web scraping and content extraction service that converts web pages (even javascript-heavy webapps) into clean markdown — the native language of LLMs. It exposes an MCP server with tools for scraping single pages, batch processing, site crawling, and web search, so an LLM agent can pull in web content as context.
Its main features AFAICT
- converting challenging JS webapps into clean markdown
- a pricing scheme that is both inconvenient and extortionate relative to its competitors—it’s an expensive monthly subscription, whereas everyone else below will take pay-as-you-go money
3 Brave
Brave Search API is search over Brave’s bespoke, in-house index. They operate their own web crawl and top it up with opt-in browsing signals from Brave browser users. Privacy claims are good: they assert no query is ever passed on to an incumbent like Google. Those major players are, by the way, retiring their search APIs, so even if we are happy with the leakage, there might still be no incumbent APIs to fall back to: Microsoft retired the Bing Search APIs, and Brave says Google’s Custom Search JSON API shuts down in 2027.
The agent-facing bit is an llm/context endpoint, from which an agent can request ranked, pre-extracted passages (text, tables, code) with source URLs and fetch timestamps. Moreover, we can filter them to a supplied token budget and relevance threshold. There are also plain web, news, image, and video endpoints, an OpenAI-compatible answers endpoint, and Goggles, which are user-defined re-ranking and domain-filtering rules.
It costs USD5 per thousand requests, with USD5 of free credit each month. Credits are available on a prepaid plan. That is competitive, insofar as many competitors only offer an inconvenient and expensive subscription model. Their marketing page cites an AIMultiple comparison that puts Brave ahead of Firecrawl, Exa, and Tavily on agent answer quality and latency. YMMV.
There is an official MCP server:
4 Kagi
Kagi, the paid ad-free search engine, now sells API access on pay-as-you-go terms: they bill every 30 days, or sooner once usage reaches $100.
searchis $12 per thousand requests, covering web, images, news, videos and podcasts. It inherits the account’s settings, so the domains I have blocked or promoted in my own Kagi searches apply to the agent’s searches too.extractis $4 per thousand pages, returning clean Markdown for up to 10 URLs per request. This competes with Jina Reader rather than with Brave or Tavily. I do not know whether it renders JS-heavy pages.
Kagi’s search results are a blend. Their own indexes (Teclis for the web, TinyGem for news) contribute the small-web material, but a typical query also fans out to a dozen or so sources, including anonymized calls to “all major search result providers”.
5 Tavily
Tavily is search-forward rather than scrape-forward, like Jina. It bills itself as a search engine for agents and RAG: a single API exposing search, extract, crawl (graph-based parallel traversal), map, and a bundled research endpoint. The search call returns relevance-scored, pre-chunked snippets sized for a model’s context window — plus an optional, synthesized answer with citations. Probably due to early-mover advantage, it is baked into many toolkits (Langchain) and cited in many papers. Pay-as-you-go is $0.008 per credit, and a basic search costs one credit, so about $8 per thousand searches.
6 Where the query goes
It has been put to me that Kagi and Brave have the most credible privacy guarantees of this bunch. Let us unpack the threat model a little though. Every vendor here sees the query text. The question, rather, is
- who else sees it, and
- whether it can be associated with me.
- Brave
- Searches over its own index. It does not fan out to other search providers. They call this “architectural” zero data retention. Moreover, they promise that queries are not shared or trained on.
- Kagi
- promises not to store my queries in a way that identifies me. They mention keeping them anonymized for up to 90 days for logistical reasons. They do pass on the queries to upstream providers, thus anonymizing my queries, but not obfuscating the content.
For Jina, Firecrawl, Tavily, Exa and Linkup I do not know where a search query goes after it leaves my machine; presumably it goes to all the other search providers.
7 And still more machine search
Many more! All the offerings seem to have converged on similar affordances now. Firecrawl now has its own search endpoint that finds pages and returns their full markdown in one call; Jina has s.jina.ai for search, plus DeepSearch, an iterative search-read-reason loop behind an OpenAI-compatible endpoint. Tavily and Brave’s llm/context are more pre-processed, trading efficiency for precision and/or control. And that’s just the start. There are ever more things!
