Search & fetch APIs for AI agents

What an agent actually needs

Strip away the marketing and an agent's web toolkit is two calls:

1. Search: turn a question into a short list of URLs and snippets. Google coverage matters because that is what users expect the agent to "know".
2. Fetch: turn a URL into clean text the model can read. Markdown beats raw HTML because it spends far fewer tokens of the context window on markup.

Everything else is a trade-off between price and capability. A fetch that renders JavaScript and rotates premium proxies gets through more pages, but costs 5–25 times the credits of a plain fetch at ScrapingBee and ZenRows. A plain residential fetch like PacketStream's returns clean Markdown from pages served as HTML, such as documentation, blogs and news, at a flat $2.50 per 1,000 with every error free. Agent traffic is also bursty: a developer may make a few hundred calls one week and none the next, which is why we priced two workloads and look closely at billing terms.

Setup notes

Hosted servers are the easy path: there is nothing to install and the vendor handles updates. PacketStream documents a one-line setup for Claude Code (its docs also cover Codex CLI, Cursor, VS Code and Claude Desktop):

export PACKETSTREAM_API_KEY="YOUR_API_KEY"
claude mcp add --transport http --scope user packetstream https://api.packetstream.io/v1/mcp \
  --header 'Authorization: Bearer ${PACKETSTREAM_API_KEY}'

The others, per their docs:

• Bright Data: an MCP server with search, scrape-as-Markdown and browser tools, available hosted (remote) or self-hosted; every new account includes 5,000 free requests a month. Pricing beyond that is not stated in the docs overview, and Bright Data's account verification is strict.
• ZenRows: https://mcp.zenrows.com/mcp over streamable HTTP with OAuth 2.1 or your API key as a Bearer token, or the local @zenrows/mcp npm package. Its docs cover Claude Code, Cursor, Windsurf, VS Code, Zed and JetBrains IDEs.
• ScrapingBee: https://mcp.scrapingbee.com/mcp?api_key=YOUR_API_KEY. The key sits in the URL, so treat the whole URL as a secret and keep it out of committed config files.
• Firecrawl: https://mcp.firecrawl.dev/v2/mcp with an API key, or /v2/mcp-oauth for browser sign-in; a keyless mode with daily limits is available for trying it out.
• Crawlbase: runs locally with npx @crawlbase/mcp and your Crawlbase tokens in the client config.

Not covered yet: ScraperAPI also documents an MCP server (hosted, plus an MIT-licensed local version), but we could not confirm how its MCP calls are billed, so it is left out of this guide until we can. Its Google results cost 25 credits on the API, about $12.25 per 1,000 on the $49 Hobby plan.

Whatever you pick, keep the API key in an environment variable, not in a project file your agent can read and echo back, and watch your usage dashboard for the first week: agents can loop.

Limits that bite

• Bright Data: MCP pricing beyond 5,000 free requests a month isn't stated, and the strictest KYC we cover applies to the account. See the Bright Data review and its SERP API pricing.
• ScrapingBee: subscription credits expire monthly; a shared global concurrency limit can return HTTP 429 under heavy load. See the ScrapingBee review.
• Firecrawl: its scraping engine came last (33.69%) in Proxyway's 2025 protected-site benchmark, and 403/404 pages still cost a credit. See the Firecrawl review.
• ZenRows: no search tool, and success fell from 70.39% at 2 requests/second to 31.76% at 10 in Proxyway's test because of plan concurrency limits. See the ZenRows review.
• Crawlbase: no search tool and no hosted server; difficulty multipliers up to ×20 apply to hard domains. See the Crawlbase review.

A practical pattern: pair a fetch-only server you like (ZenRows has the broadest set of browser-control tools) with a pay-per-call search API, so you are not paying a subscription for search you rarely use.