Open-source web scraping tools

Plugging in a rotating residential proxy

Every residential provider in our proxy guide works the same way from your code's point of view: one gateway host and port, plus a username and password. Rotation happens at the gateway; targeting and sticky sessions are usually encoded in the credentials. The examples below read everything from environment variables so you can swap providers without touching code, and so credentials never land in your repository.

For PacketStream specifically, its docs give the gateway as proxy.packetstream.io with HTTPS on port 31111 (recommended), HTTP on 31112 (legacy) and SOCKS5 on 31113. Country targeting and sticky sessions are suffixes on the auth key (your password): _country-US, _session-ID, or both. It picks a new exit on each new connection, so clients that keep connections alive keep their exit. PacketStream also publishes setup guides for Scrapy, Playwright and Puppeteer, plus code for Python, Node, Go and curl. Use whichever scheme your client supports; the PacketStream review has the details.

Scrapy

Scrapy is the purpose-built Python crawler: fast, asynchronous and extensible through middleware and item pipelines. It downloads HTML without a browser, which is exactly why it is cheap on bandwidth, and exactly why it cannot see content that JavaScript draws. Add scrapy-playwright for rendered pages or route hard ones through Zyte API (Zyte also sells Scrapy Cloud hosting from $9 per unit per month; see the Zyte review).

Proxies go through the built-in HttpProxyMiddleware. The simplest setup is environment variables:

# Every request goes through the gateway; HttpProxyMiddleware reads these.
export https_proxy="http://USER:PASS@gateway.example.com:PORT"
export http_proxy="$https_proxy"
scrapy crawl prices

Or set the proxy per request, which takes precedence over the environment:

import os
import scrapy

PROXY = os.environ["PROXY_URL"]  # http://USER:PASS@gateway.example.com:PORT

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalogue/"]

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            # Per-request proxy: overrides the environment variables
            yield response.follow(href, self.parse_product, meta={"proxy": PROXY})

    def parse_product(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}

Scrapy's documentation notes that SOCKS proxy URLs (socks5://user:pass@host:port) work only with the HttpxDownloadHandler; the default handlers don't support them. Use your provider's HTTP(S) port unless you have switched handlers. There is no built-in rotation across several proxy URLs, which is fine with a rotating gateway. Scrapy proxy docs.

Playwright

Playwright drives Chromium, Firefox and WebKit from several languages, and it is the tool to reach for when a page only makes sense after JavaScript runs, or when you need to log in, click and scroll. It is not a crawler: queueing, retries, deduplication and storage are yours to build (or let Crawlee do it). Proxies can be set per browser or per context, over HTTP(S) or SOCKS5, and page.route() lets you abort requests you don't need:

import { chromium } from 'playwright';

const browser = await chromium.launch({
  proxy: {
    server: process.env.PROXY_SERVER, // e.g. http://gateway.example.com:PORT
    username: process.env.PROXY_USER,
    password: process.env.PROXY_PASS,
  },
});
const page = await browser.newPage();

// Don't pay for bytes you won't parse: drop images, fonts, CSS and media.
await page.route('**/*', (route) =>
  ['image', 'font', 'stylesheet', 'media'].includes(route.request().resourceType())
    ? route.abort()
    : route.continue(),
);

await page.goto('https://example.com/');
console.log(await page.title());
await browser.close();

Blocking those resource types is the single biggest saving in browser scraping: a full render can pull around 2 MB, a render without assets is closer to 500 KB. Playwright network docs.

Puppeteer

Puppeteer is the lighter, Chrome-focused option for Node.js. The proxy goes in as a Chrome launch flag and the credentials through page.authenticate(), which, per its docs, turns on request interception behind the scenes and may affect performance:

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({
  args: [`--proxy-server=${process.env.PROXY_HOST}:${process.env.PROXY_PORT}`],
});
const page = await browser.newPage();
// Answers the proxy's auth challenge (turns on request interception).
await page.authenticate({
  username: process.env.PROXY_USER,
  password: process.env.PROXY_PASS,
});
await page.goto('https://example.com/');
await browser.close();

A caution on stealth: puppeteer-extra-plugin-stealth, the usual answer to "how do I avoid detection", has had no release since 1 March 2023. page.authenticate() docs.

Crawlee

Crawlee, maintained by Apify under Apache-2.0 and available for JavaScript/TypeScript and Python, wraps Playwright, Puppeteer and plain HTTP crawling in one library with queues, a session pool and proxy management. That is why it is our pick for browser crawling: the plumbing you would otherwise write around Playwright is already there. ProxyConfiguration rotates through your proxy URLs, sessions keep cookies and an IP together, and playwrightUtils.blockRequests() drops CSS, images, fonts and similar files by default (Chromium only; it never blocks the main document):

import { PlaywrightCrawler, ProxyConfiguration, Dataset, playwrightUtils } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [process.env.PROXY_URL], // http://USER:PASS@gateway.example.com:PORT
});

const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  useSessionPool: true,
  persistCookiesPerSession: true,
  preNavigationHooks: [
    async ({ page }) => {
      await playwrightUtils.blockRequests(page); // CSS, images, fonts... (Chromium only)
    },
  ],
  async requestHandler({ request, page, enqueueLinks }) {
    await Dataset.pushData({ url: request.loadedUrl, title: await page.title() });
    await enqueueLinks();
  },
});

await crawler.run(['https://example.com/catalogue/']);

With a single rotating gateway, rotation happens at the provider. If you want each Crawlee session to hold one exit IP, build the proxy URL per session with newUrlFunction:

// One sticky exit per Crawlee session, for providers that encode the session
// in the credentials (PacketStream documents a _session-<ID> suffix on the auth key).
const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: (sessionId) => {
    const label = String(sessionId).replace(/[^a-zA-Z0-9]/g, '');
    return `http://${process.env.PROXY_USER}:${process.env.PROXY_KEY}_session-${label}@${process.env.PROXY_HOSTPORT}`;
  },
});

Crawlee proxy management docs · blockRequests() reference. Prefer it hosted? Crawlee comes from Apify, whose platform can run it for you.

Self-hosted Firecrawl

Firecrawl's core is AGPL-3.0 and self-hostable, which gives you its LLM-ready Markdown and JSON output without per-page credits. Two caveats: AGPL obligations apply if you modify it and offer it as a network service, and the hosted product's results on protected sites were the weakest in Proxyway's 2025 benchmark (33.69%), so you still need your own proxies and unblocking for hard targets. The hosted service is reviewed in the Firecrawl review and covered in the AI-agent search and fetch guide.

Where DIY stops working

Open-source tools plus residential proxies are the cheapest way to scrape lightly protected sites at volume. They stop being cheap against serious anti-bot systems. ScrapeOps' June 2026 Cloudflare tests found self-hosted stealth browsers succeeding 0–35% of the time, against about 97% for scraping APIs. The usual answer is a hybrid: keep your crawler, and send only the hard domains through an unblocker from our scraping API guide. The build vs buy guide does the maths.