TL;DR

Your site is being scraped. So is every site. The useful question isn't whether, it's which scrapers, at what cost to you, and does any of it justify a response.

Three tiers, and conflating them is the standard mistake:

  • Beneficial — Googlebot, Bingbot, and the AI crawlers you want citing you. Blocking these costs you traffic.
  • Neutral — price monitors, research crawlers, SEO tools, archive services. They consume some bandwidth and don't harm you.
  • Harmful — content mirrors that republish your work, aggressive crawlers that degrade performance, and anything harvesting personal data from your users.

Detection is straightforward from server logs: group by user agent and IP range, look at request rate and path patterns, and compare against your human traffic baseline. The tells are consistent — no static assets requested, perfectly regular intervals, sequential URL walking, and no JavaScript execution.

Response should be proportionate, and the honest ranking is: robots.txt for the compliant, rate limiting for the rude, blocking for the harmful, and DMCA takedowns for actual content theft. Aggressive blanket blocking mostly costs you legitimate traffic and search visibility.

We build ScrapeMaster, so we're writing this from an unusual position. Our view is consistent in both directions: public data at a reasonable rate is fine, and site owners are entitled to defend against load and content theft. Both things are true.

What your logs actually tell you

Everything you need is in standard access logs. No special tooling required.

Signal 1: Assets versus HTML

A human browser requesting a page also requests its CSS, JavaScript, fonts, and images. A simple scraper requests only the HTML.

Compute the ratio of HTML requests to asset requests per client. Humans sit somewhere around 1:15 to 1:40. A client at 1:0 is automated, unambiguously.

Headless-browser scrapers do load assets, so this catches the simple majority, not everything.

Signal 2: Request timing

Humans are irregular. They read, they pause, they open three tabs at once, they leave for coffee.

Automation is regular. Requests at 1.00-second intervals, or in tight bursts followed by exact gaps, are machine-paced. Compute the standard deviation of inter-request intervals per client — low variance is the tell.

Signal 3: Path traversal patterns

Humans navigate by interest: a category, a product, back, a different product, search, elsewhere.

Scrapers walk systematically — sequential IDs, alphabetical listings, every page of paginated results in order, or a clean breadth-first sweep of your sitemap. A client hitting /product/1001 through /product/1400 in order is not shopping.

Signal 4: User agent honesty

Three groups:

  • Declared bots — Googlebot, bingbot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, and the rest. They say what they are. Verify the big ones by reverse DNS, since the string is trivially forged.
  • Generic clients — python-requests, curl, Scrapy, axios, default Go-http-client. Automated and not pretending otherwise.
  • Spoofed browsers — a current Chrome UA string with no asset requests and machine-regular timing. The mismatch is the signal.

Signal 5: Referrer and entry behaviour

Humans arrive from search, social, or a link, and usually enter at a small set of pages. Scrapers arrive with no referrer, at arbitrary depth, often straight into paginated listings nobody links to.

Signal 6: Concurrency from one network

Twelve simultaneous connections from a single IP or a tight IP range, sustained, is not a person. Datacenter ASNs are worth flagging in particular — residential proxies are the counter, and they're harder to spot.

A concrete detection pass

Start with the crude version. It's usually enough.

# Top user agents by request count
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30

# Top IPs by request count
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -30

# For a suspect IP: what did it request, and in what order?
grep "^203.0.113.45" access.log | awk '{print $4, $7}' | head -50

Read the third output. If it's sequential paths at regular timestamps with no assets, you have your answer in about ninety seconds.

For anything ongoing, put logs somewhere queryable — GoAccess for a quick visual pass, or your CDN's analytics if you're behind one. Cloudflare, Fastly, and similar surface bot classification directly and save most of this work.

Which of it actually matters

Leave these alone

Search engine crawlers. Blocking Googlebot removes you from Google. This is more common as an accident than you'd hope — an overzealous WAF rule or a misconfigured robots.txt can do it, and nobody notices until traffic drops.

AI crawlers, probably. This is a real decision rather than an obvious one. GPTBot, ClaudeBot, and PerplexityBot consume your content, and some of them drive citations and referral traffic back. Given that ChatGPT reportedly commands the overwhelming majority of AI referral traffic, blocking it forfeits that channel entirely. Whether the trade is worth it depends on your business model — a publisher monetizing pageviews sees this differently from a SaaS company wanting to be recommended.

Whatever you decide, decide it deliberately and write down why. The worst version is blocking by default and discovering a year later you're invisible in the surface everyone now uses.

Archive services. The Internet Archive and similar preserve your content. Rarely worth blocking.

Low-volume research and price crawlers. A few hundred requests a day costs you nothing measurable.

Watch these

Aggressive crawlers. Anything generating enough load to affect response times for real users. Rate limit rather than block — a well-behaved crawler will back off on 429s, and you find out quickly which ones are well-behaved.

Full-catalogue extraction. A competitor pulling your entire product database daily is competitive intelligence. It's generally lawful, and you're within your rights to make it expensive. Rate limiting is the proportionate response.

Act on these

Content mirrors. Sites republishing your articles verbatim. This is a copyright matter with an established remedy: DMCA takedown to the host and to Google, plus a rel=canonical on your originals. Find them by searching a distinctive sentence from your content in quotes.

Personal data harvesting. If your site exposes user profiles, comments with names, or a member directory, and something is systematically collecting it, you have a GDPR/CCPA problem of your own — you're the controller of that data, and its bulk extraction is a security consideration you're expected to have thought about.

Anything probing for vulnerabilities. Requests for /wp-admin, /.env, /.git/config, or /admin.php on a site that has none of those aren't scraping. That's reconnaissance, and it belongs in a different response process.

Responses, ranked by cost-effectiveness

1. robots.txt

Free, instant, and honoured by every crawler that cares about its reputation — which includes all the major search and AI crawlers, because being caught ignoring it is a liability for them.

User-agent: GPTBot
Disallow: /

User-agent: *
Crawl-delay: 5
Disallow: /internal/

It is not an access control. It's a request. Anything malicious ignores it. Its value is that it cleanly handles the compliant majority, and the June 2026 rulings touching robots.txt and DMCA reinforced that it's a signal with legal weight in some contexts even though it isn't a technical barrier — we covered those.

2. Rate limiting

The best value in this list. It preserves access for everyone while capping the cost of any single client.

Set a threshold well above genuine human peak — say 60 requests per minute per IP for a content site — and return 429 with a Retry-After header. Well-behaved crawlers back off. Poorly-behaved ones reveal themselves by not backing off.

Rate limiting has a large advantage over blocking: it fails safe. A false positive means someone waits, not that they're locked out.

3. CDN bot management

If you're behind Cloudflare, Fastly, or similar, their bot classification is better than anything you'll build, because it's informed by traffic across a large fraction of the web. Cloudflare shipped agent-crawler controls in 2026 that give per-category decisions rather than a blanket toggle — more here.

The trade-off is that aggressive settings produce challenges for real users, particularly those on VPNs, older browsers, or assistive technology. Tune it and watch your conversion metrics, not just your bot metrics.

4. Authentication

The only reliable boundary. If data genuinely must not be collected in bulk, put it behind a login.

This is also the legally significant line: scraping public data is well-established as lawful, while circumventing an access control is where CFAA exposure begins. Requiring authentication converts an unwinnable technical argument into a clear legal one.

5. DMCA takedowns

For actual content theft, not for scraping. Notice to the host, notice to Google for search de-indexing. Effective and well-worn.

Note the limit the July 2026 Google v. SerpApi dismissal reinforced: the DMCA's anti-circumvention provisions only apply where a technical barrier protects genuinely copyrighted material. Search results, URLs, snippets, and factual data are publicly accessible facts that copyright doesn't reach. Copyright protects your articles and images; it doesn't protect your prices, stock levels, or listings. Reaching for the DMCA over factual data is a losing move — the reasoning is here.

What generally isn't worth it

Blanket IP blocking. Whack-a-mole against anyone with proxies, and it catches real users sharing NAT'd addresses.

Heavy JavaScript obfuscation. Breaks accessibility, hurts SEO, and is defeated by a headless browser in an afternoon.

Aggressive CAPTCHA everywhere. Real cost to real users, particularly disabled ones, for modest deterrence.

Legal threats over public factual data. The case law is not on your side, and a cease-and-desist that gets publicly posted is a bad week.

The honest position

We make a scraping tool and we're not going to pretend that's a neutral vantage point.

Where we land: collecting publicly displayed data at browsing speed is legitimate, and site owners are entitled to protect against load and content theft. These coexist. A tool that reads a public product listing at human pace is not the thing degrading your servers; that's an industrial crawler with a proxy pool.

ScrapeMaster is deliberately built to sit on the acceptable side. It runs in the user's browser at browsing speed using their normal session, and it does not rotate proxies, spoof fingerprints, or solve CAPTCHAs. Those features exist specifically to defeat the defences described above, and leaving them out is a design choice.

If you're a site owner reading this: the tools that will actually hurt you are not the ones running in someone's browser tab. Spend your effort on rate limiting and on finding content mirrors, and don't let anti-scraping zeal cost you search traffic.

Frequently asked questions

How do I know if my website is being scraped?

Check your access logs for clients that request HTML but no CSS, JavaScript, or images; that request at highly regular intervals; that walk paths sequentially; and that arrive with no referrer. Grouping requests by user agent and IP, then inspecting the request sequence of the top clients, usually answers it in a couple of minutes.

Is it illegal for someone to scrape my website?

Generally not, for publicly displayed data. Courts in the US, EU, and UK have consistently found that collecting public data doesn't by itself break the law. It becomes actionable when someone circumvents authentication, republishes copyrighted content, harvests personal data without a lawful basis, or generates enough load to disrupt your service.

Should I block AI crawlers like GPTBot and ClaudeBot?

It's a genuine business decision, not an obvious yes. They consume your content and some drive citations and referral traffic back — and AI referral traffic is now concentrated enough that blocking forfeits a real channel. Publishers monetizing pageviews often decide differently from SaaS companies wanting to be recommended. Decide deliberately and record the reasoning.

Does robots.txt stop scrapers?

It stops compliant ones, which includes every major search and AI crawler, because ignoring it publicly is a reputational liability for them. It's a request, not an access control, so anything malicious ignores it. Its value is cheaply handling the well-behaved majority.

What's the most effective way to reduce scraping load?

Rate limiting. Set a threshold above genuine human peak and return 429 with Retry-After. It caps any single client's cost without locking anyone out, and it fails safe — a false positive means someone waits rather than being blocked. It also reveals which crawlers are well-behaved, since those back off.

Someone is republishing my articles. What do I do?

That's copyright infringement, not a scraping problem, and it has an established remedy: DMCA takedown to the host and to Google for de-indexing, plus rel=canonical on your originals. Find mirrors by searching a distinctive sentence from your content in quotes.

Can I use the DMCA to stop scraping of my prices or listings?

No. The July 2026 Google v. SerpApi dismissal reinforced that DMCA anti-circumvention only applies where a technical barrier protects actual copyrighted material — and prices, listings, and factual data are publicly accessible facts copyright doesn't cover. Copyright protects your articles and photography, not your catalogue data.

How do I tell a spoofed Googlebot from the real one?

Reverse DNS on the requesting IP, then forward-resolve the result to confirm it maps back. Genuine Googlebot resolves within Google's own domains. The user agent string alone is worthless — anyone can send it.

Bottom line

Your site is scraped constantly, most of it costs you nothing, and some of it — search crawlers, and possibly AI crawlers — is actively how people find you. Detection is a log-reading exercise you can do in an afternoon: assets-to-HTML ratio, timing regularity, path traversal, user agent honesty.

Then respond in proportion. robots.txt for the compliant, rate limiting for the rude, authentication for anything that genuinely must not be public, DMCA for real content theft. Skip the blanket blocking, the obfuscation, and the legal threats over public facts — those cost you more than they cost anyone else.

We build ScrapeMaster, which runs in your browser at browsing speed with no proxy rotation and no CAPTCHA solving, because the line between legitimate collection and abuse is real and worth staying on the right side of — from both directions.

If you'd rather archive your own pages than defend them: Convert: Web to PDF captures them locally as real PDFs. And CineMan AI puts IMDb and Rotten Tomatoes ratings on Netflix, Prime Video, and Disney+.