GUIDE

Build vs buy a web scraper

Running your own scraper looks free until you add bandwidth, retries, CAPTCHAs and the hours you spend fixing it. Paying an API looks expensive until you count those. Here is the arithmetic with October 2026 prices.

The short answer

Build (DIY)

Lightly protected targets, HTML-only or asset-blocked fetches, high volume, or when you need full control of the browser. Scrapy or Playwright plus a rotating residential proxy.

Buy (API)

Targets behind DataDome, Akamai, Kasada or Cloudflare; low-to-medium volume; small teams; when you would rather pay only for pages that arrive.

Hybrid

Your own crawler for the easy majority, an unblocker for the few domains that fight back.

What building really costs

1. Bandwidth

Residential proxies bill per GB, so the formula is page size × pages × price per GB. Page size is the variable people underestimate. The HTTP Archive's 2025 Web Almanac puts the median home page at 2,862 KB on desktop and the median inner page at 1,963 KB, while the HTML document alone is about 20–22 KB. The Web Scraping Club works with about 100 KB for an HTML-only fetch and about 1.7 MB for a full render. Blocking images, fonts and CSS is the main lever: a render with assets blocked is closer to 500 KB.

Prices span almost an order of magnitude: $0.99/GB (Evomi Core) and $1/GB (PacketStream) at the bottom, a $4/GB market median, $8/GB list at Bright Data. Compare them in the proxy guide.

2. Retries

Failed requests still use bandwidth. Proxyway's 2026 research found a median 74.43% success rate for residential proxies on Amazon, Google and Instagram; retrying the failures means up to about 1.34× the traffic. On Cloudflare sites it gets worse: ScrapeOps' June 2026 tests found self-hosted stealth browsers succeeding only 0–35% of the time.

3. CAPTCHAs

If your target shows CAPTCHAs, solving services charge per thousand: 2Captcha lists reCAPTCHA v2 at $1–$2.99 per 1,000 and Cloudflare Turnstile at $1.45 per 1,000. On a protected site that challenges every session, that can cost more than the bandwidth.

4. Servers and your time

Servers depend on your stack; headless browsers need far more memory than HTTP crawlers. Maintenance is the hidden line: the Web Scraping Club estimates about 30 minutes a month per simple scraper in 2026, down from about two hours in 2023. Multiply that by the number of sites and by what an hour of your time costs.

And the trend is against DIY: ScrapeOps (a vendor, so read it as such) reported in October 2025 that the share of requests needing residential IPs, JavaScript or anti-bot handling had risen from under 2% to about 25%.

What buying really costs

Scraping APIs charge per successful request, with the price set by what the request needed. Plain HTML costs 13–28 cents per 1,000 at Zyte's easiest tier, ScrapingBee and ZenRows; JavaScript rendering about $1–$5 per 1,000 on entry plans (more on Zyte's harder tiers); premium proxies plus rendering $4.90 (ScrapingBee) to $12.25 (ScraperAPI) on entry plans; Bright Data's Web Unlocker is a flat $1.50 whatever the page needs. Credit-based plans hide this behind multipliers. Proxyway's 2025 report found top configurations cost up to 100× the base rate. The scraping API guide has the full table and the credit maths.

What you get for that: no retries on your bill (most APIs charge successes only), no CAPTCHA bill, no unblocking code to maintain. What you give up: control over the browser, and predictability when an API assigns your target a pricier tier. Subscriptions also carry a monthly minimum whether or not you use the credits.

Worked example A: an easy catalogue

500,000 product pages a month from a lightly protected shop, HTML only (about 100 KB a page), near-100% success:

Verdict: close on paper. Cheap bandwidth wins by a little, but add a server and an hour of maintenance and an easy-tier API is competitive. Pick DIY if you already run crawlers; buy if you don't want to.

Worked example B: a protected site

50,000 pages a month from a site behind a serious anti-bot vendor, needing a headless browser (assets blocked, about 500 KB a page):

Verdict: the bandwidth bills look similar, but the DIY numbers assume you can reach those success rates at all, and they exclude CAPTCHA solving and the debugging time. The Web Scraping Club's estimate is that DIY only pays off on hard anti-bot sites above roughly 100,000 requests a month. Below that, buy.

The hybrid most teams end up with

Most real scraping portfolios are mixed: dozens of easy sites and a few hard ones. Keep your own crawler and residential bandwidth for the easy majority and send only the domains that keep failing to an unblocker. The routing logic is a few lines: 

from urllib.parse import urlparse
import os
import requests

PROXIES = {"http": os.environ["PROXY_URL"], "https": os.environ["PROXY_URL"]}
HARD_DOMAINS = {"www.protected-shop.example"}  # the sites that keep blocking you

def fetch(url: str) -> str:
    if urlparse(url).hostname in HARD_DOMAINS:
        return fetch_via_unblocker(url)  # your scraping API client: billed per success
    r = requests.get(url, proxies=PROXIES, timeout=30)  # residential bandwidth: billed per GB
    r.raise_for_status()
    return r.text

Track success rate per domain and move domains between the two lists as their protection changes. Our cost calculator compares both sides for a given volume, and the open-source guide shows how to wire proxies into Scrapy, Playwright, Puppeteer and Crawlee.

Decision checklist

  1. Do your targets block plain requests from datacenter IPs? If not, you may not need residential proxies at all.
  2. Do the pages need JavaScript? If not, an HTTP crawler at 100 KB a page is the cheapest option there is.
  3. Which anti-bot vendor protects each target? DataDome, Akamai, Kasada and Cloudflare usually tip the answer to buy.
  4. How many pages a month? Below roughly 100,000 on hard sites, buying usually wins.
  5. Who maintains it? If nobody owns the scraper, buy, or use a no-code tool.
  6. Is the data for an LLM? Look at search and fetch APIs with MCP servers.

Sources