Web search for machines

Search and scraping APIs for agents, which do not click links

2026-07-24 — 2026-07-24

agents
computers are awful together
NLP
provenance
search
wonk
Figure 1

An internet search engine for humans returns an HTML page of links, on the reasonable assumption that a human will click a few of them to read. An agent does not click. It would much rather not parse the HTML into text, stripped of nonsense, chunked, and ideally carrying enough provenance that the model can say where a claim came from. There is a whole market of vendors selling that AI-facing product now.

For the retrieval machinery underneath — chunking, embeddings, BM25, reranking — see AI search.

1 Jina

Below I mention Firecrawl as a web scraping tool. It is kind of the default because of being famous etc.

However it is too gold-plated for me. As far as I can tell, they charge a premium for using browsers for scraping, which is a good way to get clean content from dynamic JS-first websites, but it’s also slow and resource-intensive and overkill for 80% of my use cases.

Jina is way cheaper.

claude mcp add -s user --transport http jina https://mcp.jina.ai/v1 \
  --header "Authorization: Bearer ${JINA_API_KEY}"

2 Firecrawl

Firecrawl is a web scraping and content extraction service that converts web pages (even javascript-heavy webapps ) into clean markdown — the native language of LLMs. It exposes an MCP server with tools for scraping single pages, batch processing, site crawling, and web search, so an LLM agent can pull in web content as context. Supports both cloud and self-hosted deployment (of the MCP server, I mean, not the internet-scale scraping).

Its main features AFAICT

  • converting challenging JS webapps into clean markdown
  • charging extortionately for it relative to its competitors

3 Tavily

Tavily is search-forward rather than scrape-forward like Jina. It bills itself as a search engine for agents and RAG: a single API exposing search, extract, crawl (graph-based parallel traversal), map, and a bundled research endpoint. The search call returns relevance-scored, pre-chunked snippets sized for a model’s context window — and optionally a synthesized answer with citations. Their argument framing is that an agent will proceed more efficiently with grounded facts-plus-provenance, rather than search results that are just a list of links to follow. For that reason, plus maybe early-mover advantage, it seems to be baked into many toolkits (Langchain) and cited in many papers.