Most "best web scraping tools" lists mix Chrome extensions, enterprise proxy networks and B2B databases into one ranking, which doesn't help anyone choose. This one is narrower. It's for people who build things: developers, data engineers and GTM engineers who need scraped data to flow into a pipeline on a schedule, from sites that render with JavaScript and push back against bots.

If you're after point-and-click tools and browser extensions for one-off extraction, our broader guide to tools to extract data from a website covers those in depth. Here we focus on the ten frameworks and scraping services that hold up in production, what each is actually good at, and when you shouldn't be scraping at all.

Every figure here came off the vendor's own pricing page, read in September 2026. Scraping prices move with the arms race, so re-read them on the day you commit.

The 10 tools at a glance

Prices change often in this category, so check each vendor's pricing page before you budget.

Four questions that narrow the choice

• Is the data in the HTML, or rendered by JavaScript? Open the page, view source, and search for the value you want. If it's there, a plain HTTP request and a parser will do, and they're far cheaper and faster to run than a browser. If it isn't, check the network tab first: many "JavaScript-heavy" sites load their data from a JSON endpoint you can call directly. Only reach for a headless browser when neither works.

• Will the site block you? A few hundred pages from a small site, probably not. Thousands of pages a day from an e-commerce or travel site, almost certainly. Blocking is what pushes teams from open-source code to paid APIs and proxy networks.

• Who maintains it? Every scraper breaks when the target site changes. If there's no engineer to fix selectors next month, prefer a managed service or a maintained marketplace scraper over custom code.

• Where does the data go next? Scraping is rarely the goal. If the output has to be cleaned, matched to companies and pushed into a CRM, plan that part before you pick the scraper.

Open-source frameworks

1. Scrapy

Scrapy is the most established Python framework for crawling at scale, maintained by Zyte and released under a BSD license. You write "spiders" that define where to start, which links to follow and how to parse each page, and Scrapy handles the rest: asynchronous requests, retries, throttling, cookies, and item pipelines that clean and store what you extract.

Its strength is crawling many pages efficiently: a whole product catalog, every job posting on a careers site, every page in a directory. It doesn't execute JavaScript by default. For rendered pages, the scrapy-playwright plugin lets specific requests go through a real browser while the rest stay fast.

Use it for: large, structured crawls you want to own and run on your own infrastructure. Skip it if: you only need a handful of pages; the framework is overkill.

2. Beautiful Soup

Beautiful Soup isn't a scraper on its own. It's a Python library for parsing HTML, usually paired with requests or httpx to fetch pages. It's forgiving with broken markup and easy to read, which is why it's the first tool most people learn. Swapping in the lxml parser speeds it up noticeably.

There's no crawling, concurrency or JavaScript rendering, so it's best for scripts: pulling a table from a page, parsing HTML you already downloaded, or processing the output of a browser tool.

Use it for: quick extraction scripts and notebook work. Skip it if: the content is rendered client-side or you need to crawl thousands of pages.

3. Playwright

Playwright, from Microsoft, has become a common default for new browser-based scraping projects. One API drives Chromium, Firefox and WebKit, with official libraries for JavaScript/TypeScript, Python, Java and .NET. Two features make it especially good for scraping: it waits automatically for elements to be ready (fewer flaky timing bugs), and it can intercept network traffic, so you can often capture the JSON a page loads instead of parsing the rendered HTML.

A minimal example in Python that loads a JavaScript-rendered page and reads product names:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    names = page.locator(".product-card h3").all_inner_texts()
    print(names)
    browser.close()

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    names = page.locator(".product-card h3").all_inner_texts()
    print(names)
    browser.close()

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    names = page.locator(".product-card h3").all_inner_texts()
    print(names)
    browser.close()

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")
    names = page.locator(".product-card h3").all_inner_texts()
    print(names)
    browser.close()

Use it for: single-page apps, infinite scroll, pages behind interactions like "load more" buttons. Skip it if: the data is in the raw HTML; a browser is much slower and heavier than an HTTP request.

4. Puppeteer

Puppeteer is Google's Node.js library for controlling Chrome. It predates Playwright (several of Playwright's original authors worked on Puppeteer) and still has a huge ecosystem of examples and plugins. Since version 23 it also supports Firefox through the WebDriver BiDi protocol, though Chrome remains its home turf.

If your team is JavaScript-only and already knows Puppeteer, there's little reason to switch. For new projects, Playwright's multi-language support and auto-waiting usually make it the easier choice.

Use it for: Chrome-focused automation in Node.js, screenshots and PDFs, existing Puppeteer codebases.

5. Selenium

Selenium is the oldest of the browser automation tools and the W3C WebDriver standard grew out of it. It supports Chrome, Firefox, Edge and Safari, with bindings for Java, Python, C#, Ruby and JavaScript, and Selenium Grid spreads sessions across many machines. Selenium Manager now downloads the right browser drivers automatically, which removed one of its old headaches.

For pure scraping it's generally slower and more verbose than Playwright. It wins when your organization already runs Selenium for testing, or when you need a language Playwright doesn't officially support, like Ruby.

Use it for: teams with existing Selenium infrastructure or skills.

Managed platforms and scraping APIs

6. Apify

Apify is a cloud platform for running scrapers, called Actors, with a large store of ready-made ones for common targets like Google Maps, e-commerce sites, search engines and social platforms. You can run a maintained Actor with a few inputs, or deploy your own code and let Apify handle scheduling, proxies, storage and retries. Its open-source library Crawlee (for JavaScript and Python) wraps Playwright, Puppeteer and plain HTTP crawling with blocking protection built in, and runs fine outside Apify too.

Pricing is usage-based on top of a free plan, and many Store Actors charge per result or by rental. Quality varies between Actors, so check how recently one was updated and its reviews before you depend on it.

Pricing: Free plan with $5 of included platform usage. Starter $19/month with $19 of usage, Scale $199/month, Business $999/month, with 10% off on annual billing. Apify states in its own pricing FAQ that unused usage credits are not rolled over to the next billing cycle and expire at the end of it (September 2026).

Use it for: getting data from popular sites quickly without writing a scraper, or hosting your own crawlers without managing servers.

7. ScrapingBee

ScrapingBee turns scraping into an API call: send a URL, get back HTML (or extracted data), with headless browser rendering and proxy rotation handled for you. It fits well when you want to keep your own parsing code but stop maintaining browsers and proxy pools.

Watch the credit math. ScrapingBee charges credits per request, and the cost depends on options: a request with JavaScript rendering costs 5 credits, while premium proxies cost more again. Turning off rendering for pages that don't need it is the easiest way to cut the bill.

Pricing: Hobby $19/month for 75,000 API credits, Freelance $49/month for 250,000, Startup $99/month for 1,000,000, Business $249/month for 3,000,000, Business+ $599/month for 8,000,000 (September 2026). At 5 credits per rendered request, the Hobby tier is 15,000 rendered pages a month, not 75,000.

Use it for: developers who want a simple, language-agnostic endpoint and moderate volumes.

8. Firecrawl

Firecrawl is built for the AI era of scraping. Give it a URL and it returns clean markdown (or structured JSON against a schema you define) instead of raw HTML, which is exactly what you want when the next step is an LLM. It can also crawl a whole site or map all of its URLs. The core is open source under AGPL-3.0 if you want to self-host, and there's a hosted API; structured extraction formats cost more credits per page than a basic scrape.

It is also the easiest of these tools to hand to an agent, because an MCP server exists and the output format needs no post-processing before it reaches a model's context.

Pricing: Free plan with 1,000 credits per month. Hobby $16/month for 5,000 credits, Standard $83/month for 100,000, Growth $333/month for 500,000, Scale $599/month for 1,000,000, all billed annually. Enterprise custom (September 2026).

Use it for: feeding websites into RAG systems, AI agents and research workflows; turning company websites into text you can classify or summarize. Skip it if: you need precise, field-level extraction from a heavily defended site at high volume.

9. Bright Data

Bright Data (formerly Luminati) is infrastructure more than a scraper. Its core is a large proxy network across datacenter, residential, ISP and mobile IPs. On top of that sit Web Unlocker (an endpoint that handles blocking, CAPTCHAs and retries for you), Scraping Browser (a hosted browser you connect Playwright or Puppeteer to), scraper APIs for specific sites, and pre-collected datasets.

Pricing varies by product: proxies are billed by bandwidth or IP, and the unblocking and scraper APIs by request, with pay-as-you-go and committed plans.

Pricing: Web Unlocker has a free tier of 5,000 requests a month. Pay-as-you-go is $1.50 per 1,000 requests with no commitment. The Scale plan is $499 a month with 383,000 requests included and $1.30 per 1,000 after that, and Enterprise is custom. Other Bright Data products are priced separately, with proxies billed by bandwidth or IP (September 2026). It gets expensive quickly at volume, so measure on a small test before committing.

Use it for: high-volume projects against sites that actively block scrapers, and geo-specific data collection.

10. SerpApi

If what you need is search results (Google organic results, Maps, Shopping, News and other engines), don't build a Google scraper. It will break constantly. SerpApi returns search results as structured JSON through a simple API, priced by monthly search volume, with client libraries for most languages. It's widely used for SEO tracking, competitive monitoring and building lead lists from search queries.

One thing to know: Google sued SerpApi in December 2025, alleging it bypassed Google's anti-scraping protections. SerpApi denies wrongdoing and the case is ongoing. If search data is business-critical for you, keep an alternative provider in mind.

Pricing: Free plan with 250 searches per month. Starter $25/month for 1,000 searches, Developer $75/month for 5,000, Production $150/month for 15,000, Big Data $275/month for 30,000, with higher tiers up to millions of searches (September 2026).

Use it for: any workflow that starts with a search query rather than a known URL.

Which of these can an agent call directly?

A growing share of scraping is triggered by an AI agent rather than a cron job, and that changes what "easy to integrate" means. An agent needs a tool it can call in one step and output it can read without a parsing layer in between.

The practical consequence: for a one-off "read this page and tell me what the company does" step inside an agent workflow, a markdown-first API is worth more than a stronger unblocker. For a nightly crawl of 200,000 defended pages, the unblocker wins and nobody cares about the output format.

When to scrape, and when to enrich instead

For GTM work in particular, a lot of scraping projects shouldn't exist. If what you need is standard B2B data (work emails, phone numbers, job titles, company size, funding, tech stack), data providers already collect and maintain it. Building a scraper for it means doing the same job worse, maintaining it forever, and often breaking a platform's terms along the way.

The split is not close. A scraper pointed at LinkedIn to find work emails will be slower, less accurate and more fragile than an enrichment call, and it puts an account at risk. A scraper pointed at a regional trade association's member directory will find something no provider sells. Knowing which side of that line you are on before you start is most of the decision.

Most real pipelines do both: scrape to find and qualify the accounts, then enrich to find the people. That's where Databar fits. Its catalog of 160+ integrations includes scraping-style enrichments (full page content via Firecrawl, Google search results, Google Maps listings, contact details published on a company's website) alongside contact and company enrichment across 100+ data sources and email waterfalls. You can run a website scrape, an AI Researcher prompt that classifies the result, and a work email waterfall as columns in the same table, and pull it all into code through the REST API, Python SDK, CLI or MCP server. Billing is outcome-based, so lookups that return nothing cost nothing, which matters on a list where half the domains are dead.

It won't replace Scrapy for crawling a million-page catalog, but for the common GTM case of "take these 2,000 domains, figure out which ones fit, and find the right person," it saves writing and maintaining the glue. If you'd rather call providers from your own code, our technical guide to data enrichment APIs compares the options, and the waterfall enrichment comparison covers the cascade side specifically.

The two-tool pattern most GTM stacks end up with

After the experimenting is done, most teams converge on the same shape: one structured data layer for companies, contacts, emails and signals, plus one page-level scraper for the unstructured edge cases. Not five scrapers competing for the same job.

The edge cases that actually justify a scraper in GTM are narrower than people expect. Niche directories that no B2B database covers, such as association member lists, regional business catalogs and conference attendee pages. Public profile pages, such as About pages, leadership teams and careers pages with open roles. And one-off competitor research, such as pricing pages, customer logos and integration partners, which an agent needs once per campaign rather than as a recurring enrichment step.

Everything else on a typical GTM list is structured data that somebody already maintains, and buying it is cheaper than owning a scraper for it.

Mistakes that break scraping projects

Scraping behind a login

Scraping public, logged-out pages and scraping pages you can only see after accepting a platform's terms are very different legally. In Meta v. Bright Data (2024), a court found Meta's terms didn't prohibit logged-off scraping of public data. In the long hiQ v. LinkedIn fight, hiQ in the end agreed to a judgment for breaching LinkedIn's user agreement and to stop scraping. Using your own LinkedIn account to scrape profiles is the classic way to lose the account. This isn't legal advice; if the data matters, get some.

No rate limiting or proxy plan

Hammering a site from one IP gets you blocked and can degrade the site for real users. Throttle requests, cache pages you've already fetched, and budget for proxies or an unblocking API from the start if the target is protected.

Brittle selectors

Selectors like div:nth-child(3) > span break on the next redesign. Prefer stable attributes, IDs and data attributes, text anchors, or structured data already in the page (JSON-LD, embedded JSON). Better still, find the API the page calls.

Silent failures

The worst scraper failure isn't a crash, it's a run that "succeeds" and returns empty fields for three weeks. Validate output on every run: row counts, required fields present, values in expected ranges. Alert when they drift.

Ignoring robots.txt and terms

robots.txt isn't legally binding everywhere, but it tells you what the site owner considers acceptable, and ignoring it weakens your position if there's ever a dispute. Read it, read the terms, and check whether the site offers an API or data export before scraping.

Collecting personal data without a plan

Names, emails and phone numbers are personal data under GDPR and similar laws, even when they're public. Know your lawful basis, collect only what you need, and have a way to delete records on request.

Budgeting on the headline credit price

Every API on this list prices in credits, and a credit rarely equals a page. ScrapingBee's Hobby tier is 75,000 credits, which is 15,000 rendered pages once JavaScript rendering takes its 5 credits per request. Firecrawl's structured extraction costs more per page than a plain scrape. Apify Store Actors often bill per result on top of platform usage. Work out cost per successfully extracted page on a small test run before you size a plan.

Also worth reading

• How to scrape Google Maps and find business owners with Claude Code

• 10 best Reddit scrapers for data extraction

• Best B2B data enrichment tools, for the half of the pipeline you should not be scraping

FAQ

What are the most popular web scraping frameworks?

In Python, Scrapy for crawling and Beautiful Soup for parsing. For browser automation, Playwright is the common choice for new projects, with Puppeteer (Node.js) and Selenium still widely used. Crawlee, from Apify, is a popular higher-level option in both JavaScript and Python.

What's the best tool for scraping JavaScript-heavy websites?

First check whether the page loads its data from an API you can call directly. If not, use Playwright or Puppeteer yourself, or a scraping API with rendering (ScrapingBee, Firecrawl, Bright Data's Scraping Browser) if you don't want to run browsers and proxies.

What's the best way to get Google search results?

Use a search results API rather than scraping Google directly, which gets blocked fast. SerpApi is the best known, starting at $25/month for 1,000 searches; Bright Data and others also offer SERP APIs, and Databar includes a Google search results enrichment you can run on a list of queries.

How much do web scraping tools cost?

The frameworks are free, and you pay in engineering time and infrastructure instead. Among the managed services in September 2026, entry tiers run from $16/month (Firecrawl Hobby) and $19/month (Apify Starter, ScrapingBee Hobby) to $25/month (SerpApi Starter). Bright Data's Web Unlocker is free for the first 5,000 requests a month and $1.50 per 1,000 after that on pay-as-you-go. What actually drives the bill is how aggressively your target sites resist, since rendering and premium proxies multiply the per-request cost.

Which web scraping tool works best with AI agents?

Firecrawl, for page-level extraction. It returns clean markdown or schema'd JSON that goes straight into a model's context without a parsing layer, and an MCP server exists so an agent can call it in one step. SerpApi is the equivalent for search queries. For the structured B2B data an agent usually also needs, an enrichment platform beats any scraper on consistency and match rate.

Should I use a scraper or a data provider?

Both, for different jobs. Data providers cover companies, contacts, emails, phones, firmographics and signals, and they maintain that data for you. Scrapers cover what no provider sells: niche directories, public profile pages, one-off competitor research. If you find yourself writing a scraper to get work emails or company headcount, you are rebuilding something you can buy.

Is web scraping legal, and do I need to code to do it?

Scraping publicly available data is often legal, but it depends on where you are, what you collect, whether you had to log in or accept terms, and how you use the data. Personal data brings privacy law into play, so check the site's terms and get legal advice for anything business-critical. As for the code, not always. Apify's ready-made Actors and search APIs need very little code, and no-code tools like Octoparse, ParseHub and browser extensions cover simple jobs. For those, see our guides to website data extraction tools and Instant Data Scraper alternatives. Firecrawl and ScrapingBee are also usable from a single HTTP call, which is within reach of anyone comfortable in a spreadsheet tool or a no-code builder.