TL;DR
"Are web crawlers legal?" and "is web scraping legal?" get conflated constantly, and they have genuinely different answers — because they're different activities with different risk profiles.
Crawling is discovery: following links to find what exists. It's how search engines work, it's expected by nearly every site, and the main legal exposure is about load — how hard you hit the server — plus whether you respected robots.txt.
Scraping is extraction: taking specific content off a page. The exposure here is about what you take and what you do with it — copyright, database rights, personal data, and contract terms you agreed to.
Most real tools do both, which is why the distinction gets lost. But it determines which questions you need to answer:
| Crawling | Scraping | |
|---|---|---|
| Main risk | Server load, trespass-style claims | Copyright, personal data, contract |
robots.txt | Directly relevant | Relevant, less decisive |
| Volume matters | A lot | Somewhat |
| What you take matters | Barely | Enormously |
| Logged-in access | Rarely involved | Often the deciding factor |
ScrapeMaster sits at the low-risk end of both: it runs in your browser, on pages you're already viewing, at the pace you browse. It doesn't crawl a site autonomously, doesn't rotate proxies or fingerprints, and doesn't bypass logins or CAPTCHAs.
General information, not legal advice. If your project has real commercial stakes, get counsel who knows your jurisdiction.
The actual distinction
Crawling
A crawler starts from a URL, fetches the page, finds the links, and follows them. Repeat. The output is a map: which pages exist, how they connect, what changed.
Googlebot, Bingbot, the Internet Archive's crawler, uptime monitors, SEO auditing tools, security scanners.
The defining property is autonomy. Nobody chooses each page. The crawler decides where to go next, which is what makes scale — and load — the central question.
Scraping
A scraper takes a page and pulls specific values out: price, name, title, review text. The output is structured data.
The defining property is selectivity. You've decided what matters and you're extracting exactly that. Which is why what you extract is the legal question, not how you got there.
Where they meet
Nearly every practical system does both. A price monitor crawls a category to find products, then scrapes each product page.
But the risks stay separable, and you should analyse them separately. A gentle crawler that scrapes copyrighted articles wholesale has a copyright problem, not a crawling problem. An aggressive crawler that extracts nothing but URLs has a load problem, not a content problem.
The legal frameworks, briefly
Computer misuse and unauthorised access
In the US, the Computer Fraud and Abuse Act. The Supreme Court's Van Buren decision (2021) narrowed "exceeds authorized access" to a gates-up-or-down question: is there a technical barrier you got past? The Ninth Circuit's hiQ v. LinkedIn line of cases established that scraping public data — no login, no barrier — doesn't violate the CFAA.
The practical rule that survives all of it: if you had to defeat a technical access control — a login, a paywall, a CAPTCHA, an IP block you evaded — you've moved into materially riskier territory. If the data was served to anyone who asked, you haven't.
The UK's Computer Misuse Act and comparable laws elsewhere land in a similar place, though the details differ.
Copyright
Facts aren't copyrightable. Expression is.
- Prices, model numbers, addresses, dates, counts: facts. Not protected.
- Article text, photographs, product descriptions, reviews: expression. Protected.
This is the distinction that decided Google v. SerpApi in 2026, where a California federal court dismissed Google's DMCA anti-circumvention claims, reasoning that the anti-circumvention provisions protect access to copyrighted works — and facts and URLs aren't that.
The practical implication: extracting a price is a very different act from copying an article, even when the same tool does both in the same session.
Database rights (EU/UK)
A right that has no US equivalent and catches people out. The sui generis database right protects substantial investment in obtaining, verifying, or presenting a collection — even where no individual item is copyrightable.
So a European directory of factual business listings can be protected as a database even though each listing is a bare fact. Extracting a substantial part may infringe regardless of copyright.
If you're extracting at scale from EU or UK sources, this deserves real attention.
Data protection
If what you're taking identifies people, GDPR and UK GDPR apply — regardless of whether the data was public. Public does not mean unregulated.
You need a lawful basis, you must provide privacy information (including where you got the data), and you must honour objections. Twenty US states now have comprehensive privacy laws in effect, and California's DROP platform for data broker deletion requests went live on August 1, 2026.
Contract
The one people ignore and shouldn't.
Browsewrap — terms linked in a footer that you never clicked — is inconsistently enforced. Courts often find no assent.
Clickwrap — terms you actively accepted, typically at account creation — is routinely enforced.
This is why "I logged in" changes everything. By logging in you almost certainly accepted terms, and those terms very likely prohibit automated extraction. Now it's a breach of contract question, and the "it was public data" defence doesn't apply to data that isn't public.
What robots.txt actually does
Widely misunderstood in both directions.
What it is: a voluntary convention. A file at /robots.txt telling automated clients which paths the site would prefer they not fetch.
What it isn't: legally binding, on its own, in most jurisdictions. It's not a technical access control. Ignoring it isn't hacking.
Why it still matters enormously:
- Evidence of notice. If you ignored a clear directive and a dispute follows, that's a bad fact. It shifts the tone from "I didn't realise" to "I was told."
- It's referenced in terms of service. Many sites' terms incorporate robots.txt by reference, which converts a convention into a contractual obligation.
- It's cheap to comply with. The paths sites disallow are usually search endpoints, infinite calendars, and session URLs — things that waste your time anyway.
- The norm is hardening. Since the AI-training disputes began, sites have been far more deliberate about robots.txt, and courts have been more willing to treat it as meaningful notice.
For crawling, robots.txt is close to decisive. Autonomous crawling against an explicit disallow is the clearest version of the bad fact.
For scraping a handful of pages you're browsing, it's a much weaker signal — you're not operating an automated crawler, and the convention was written for ones that are.
Crawl-delay and rate
Load is the crawler-specific risk, and it's the one that produces the most real-world consequences — not because of a lawsuit, but because you get IP-blocked and your project stops.
Reasonable practice: honour Crawl-delay where specified, keep concurrency low, back off on 429 and 503, avoid peak hours, and identify yourself in the user agent if you're running anything substantial.
A crawler that degrades a site's performance can attract trespass-to-chattels claims in the US, and those claims turn on demonstrable harm — which is precisely why rate limiting is both courtesy and legal risk management.
Risk tiers
| Activity | Risk | Why |
|---|---|---|
| Manually browsing and extracting factual data from a public page | Very low | No barrier, facts, human pace |
| Extracting public factual data at browsing pace | Low | Same, structured |
| Crawling a public site politely, respecting robots.txt | Low | Expected behaviour |
| Extracting personal data from public pages | Medium | GDPR applies regardless of publicity |
| Extracting substantial parts of an EU database | Medium–high | Database right |
| Crawling aggressively, ignoring robots.txt | Medium–high | Notice + load |
| Extracting from behind a login you accepted terms for | High | Contract |
| Bypassing a paywall, CAPTCHA, or IP block | Very high | Access control defeated |
| Republishing copyrighted content wholesale | Very high | Copyright |
The gap between the top and bottom rows is not a matter of degree. Different laws apply.
Where a browser extension sits
Worth being precise about, because "is it legal" often really means "is what I'm doing the risky version."
ScrapeMaster runs inside your browser, on pages you have open, using your normal session. That places it at the low-risk end structurally:
- It doesn't crawl autonomously. You navigate; it extracts what's on screen. The load question mostly doesn't arise — you're generating the traffic a person browsing generates.
- It doesn't bypass anything. No paywall circumvention, no CAPTCHA solving, no login defeat. If you can see it, it can extract it; if you can't, neither can it.
- It doesn't rotate proxies or fingerprints. Evading a block is a meaningfully worse fact than being blocked, and the tool doesn't offer the option.
- Your data stays local. Extracted records live in your browser's IndexedDB. The only network call is during auto-detect, sending page structure rather than content.
What it doesn't do is make your use lawful. If you're logged into a site whose terms forbid automated extraction, the tool being polite doesn't change the contract you accepted. If you extract personal data, GDPR applies to you as the controller. The tool is neutral; the use is yours.
That's the honest framing, and it's the one we'd want if we were on the other side of it.
A practical checklist
Before a project:
- Public or authenticated? If you had to log in, read the terms. This is the single highest-signal question.
- Facts or expression? Prices and specs are safer ground than article text and photographs.
- Does it identify people? If yes, you need a lawful basis and a privacy notice.
- EU or UK source, at scale? Consider database rights.
- What does robots.txt say? Read it. Comply, or have a considered reason.
- What's your rate? Slow enough to be invisible is slow enough to be fine.
- What are you doing with it? Internal analysis is a world away from republishing or reselling.
- Would you defend it in writing? If describing your method to the site owner makes you uncomfortable, that's information.
Question 8 is unscientific and catches more problems than the rest.
Frequently asked questions
Are web crawlers legal?
Crawling publicly accessible pages is generally lawful — it's how search engines and archives function, and most sites expect it. The risk comes from volume and disregard: crawling hard enough to degrade a site can support trespass-style claims in the US, and ignoring an explicit robots.txt disallow is evidence you were on notice. Polite crawling of public pages that respects robots.txt sits at the low end of the risk scale.
What's the difference between a crawler and a scraper, legally?
A crawler discovers pages by following links; a scraper extracts specific content from pages. Crawling risk centres on server load and whether you respected robots.txt. Scraping risk centres on what you take — copyright in expressive content, database rights in the EU and UK, data protection where individuals are identified, and contract terms if you were logged in. Most tools do both, but the risks should be analysed separately.
Is ignoring robots.txt illegal?
Not on its own, in most jurisdictions — it's a voluntary convention, not a technical access control, and ignoring it isn't unauthorised access. But it matters: it's evidence you had notice, many terms of service incorporate it by reference (making it contractual), and courts have grown more receptive to treating it as meaningful. For autonomous crawling, treat it as close to binding.
Does logging in change whether scraping is legal?
Substantially, yes. Logging in usually means you accepted terms of service via clickwrap, which courts routinely enforce, and those terms commonly prohibit automated extraction. It also means the data isn't public, so the public-data reasoning from cases like hiQ v. LinkedIn doesn't apply. The analysis shifts from access law to contract law.
Can I scrape public data that includes people's names?
Public availability doesn't exempt personal data from GDPR or UK GDPR. You need a lawful basis (usually legitimate interests for B2B), you must provide privacy information including the source, and you must honour objections. In the US, twenty states now have comprehensive privacy laws in effect, and California's DROP deletion platform for data brokers went live on August 1, 2026.
What are database rights and do they apply to me?
A protection in the EU and UK for substantial investment in compiling, verifying, or presenting a collection — with no US equivalent. It can protect a directory of purely factual listings even though no single listing is copyrightable, so extracting a substantial part may infringe independently of copyright. If you're extracting at scale from EU or UK sources, take it seriously.
Does using a browser extension instead of a script make scraping legal?
It reduces some risks and changes none of the legal analysis of what you do with the data. Extracting at browsing pace from pages you already have open avoids load-related exposure and avoids the access-control question entirely, since nothing is bypassed. But contract terms, copyright, database rights, and data protection apply to you regardless of which tool you used.
Bottom line
Crawling and scraping get different answers because they present different risks. Crawling is about load and notice. Scraping is about what you take, and what you agreed to before you took it.
The questions that actually separate low risk from high: did you get past a technical barrier, did you accept terms by logging in, is what you took factual or expressive, and does it identify people. Everything else is detail.
The reliably safe zone is narrow and useful: public pages, factual data, human-scale pace, robots.txt respected, internal use. Most legitimate business research fits comfortably inside it.
ScrapeMaster is free, runs in your browser at your pace, doesn't bypass access controls, and keeps extracted data local. It doesn't make your use compliant — that part's yours. Our longer treatment of the case law is in is web scraping legal.
Different kind of decision paralysis: CineMan AI picks what to watch.