How to Read a Site's robots.txt and Automated-Access Terms (Amazon as a Worked Example)

TL;DR

Most arguments about whether you can extract from a site are conducted without anyone having read what the site says. The documents are public, short, and take about ten minutes.

The three you need:

1. /robots.txt — which paths automated clients are asked not to fetch
2. The terms of service / conditions of use — the contractual position, including any automated-access clause
3. Any developer or API terms — often the only sanctioned route, and often free

How to read them, in order:

• Find the User-agent: * block in robots.txt — that's the one about you
• Look for Disallow: lines covering the paths you care about
• Search the terms for: robot, spider, scraper, crawl, automated, data mining, extract, harvest
• Check whether an API exists and whether it serves your use case

This post uses Amazon as the worked example, because it's one of the most restrictive major sites and it's the one people ask about most. The method transfers to any site.

General information, not legal advice. Read the current documents yourself — terms change, and this post is a snapshot of how to read them, not a substitute for reading them.

Step 1: robots.txt

Every site serves it at the root: example.com/robots.txt. Type it in your address bar.

The structure

User-agent: <which client this block applies to>
Disallow: <path prefix not to fetch>
Allow: <exception carved out of a Disallow>
Crawl-delay: <seconds between requests>

Sitemap: <URL of the sitemap>

The critical part people miss: blocks are per-user-agent, and only the most specific matching block applies. If a site has a block for Googlebot and a block for *, Googlebot obeys its own block and ignores the wildcard one entirely — it does not merge them.

So when reading for your own purposes, find the User-agent: * block. That's the one addressing you.

What large e-commerce robots.txt files typically disallow

Read Amazon's and you'll find a long file. The pattern across large retail sites is consistent, and understanding why each category is there is more useful than memorising the list:

The pattern worth internalising: the heaviest restrictions cluster around search result pages with query parameters, anything session- or account-specific, and anything that writes. Product detail pages are often treated differently from search listing pages — which matters, because it maps directly onto the difference between "look up this specific item" and "enumerate everything in this category."

What robots.txt does and doesn't mean

It's a convention, not an access control. Ignoring it isn't unauthorised access under the CFAA or equivalent laws — the Supreme Court's Van Buren decision framed that as a gates-up-or-down question about technical barriers, and robots.txt isn't a barrier.

But it establishes notice, and notice matters. If a dispute follows, "the site published a directive and I ignored it" is a materially worse position than "I wasn't aware." Courts have grown more receptive to treating robots.txt as meaningful since the AI-training disputes began.

And it's frequently incorporated by reference into terms of service, which converts a convention into a contract term.

Step 2: The terms

Finding the right document

Large sites have several. Look for the one governing use of the site itself:

• Amazon: Conditions of Use
• Most sites: Terms of Service, Terms of Use, or Legal
• Usually linked in the footer

Beware of reading the wrong one. Seller agreements, developer terms, and privacy policies are different documents with different scope.

Searching efficiently

Cmd/Ctrl + F for: robot, spider, scraper, scrape, crawl, automated, data mining, extract, harvest, systematic, monitor, aggregat

Ten seconds and you'll find the relevant clause or establish there isn't one.

What automated-access clauses look like

Amazon's Conditions of Use contain a licence-and-access provision prohibiting things like data mining, robots, and similar data gathering and extraction tools without express written consent. That general shape — no automated extraction absent written permission — is standard across large commercial sites.

Reading these clauses properly means noticing three things:

Whether the prohibition is unconditional or qualified. Some prohibit automated access outright; others prohibit it only where it burdens the service, or only for commercial purposes. The qualifier changes the analysis substantially.

Whether an exception exists. "Except as expressly permitted" usually points somewhere — an API, a partner programme, an affiliate agreement.

How you're bound. This is the part that determines whether the clause matters much or a lot:

• Browsewrap — terms linked in the footer that you never clicked. Enforcement is inconsistent; courts often find no assent.
• Clickwrap — terms you actively accepted, typically at account creation. Routinely enforced.

So being logged in changes the position substantially. With an account, you almost certainly clicked through terms, and now the automated-access clause is a contract term you agreed to. Browsing logged-out, the position is considerably weaker.

This is the single most useful thing to understand from this whole post.

The consequence is usually contractual, not criminal

Breaching a terms-of-service clause about automated access is typically a breach of contract issue, not a computer crime one. The realistic consequences, in order of likelihood:

1. Rate limiting
2. CAPTCHA challenges
3. IP or account block
4. Account termination
5. A cease-and-desist letter
6. Litigation

Most people encounter 1–3 and never anything further. That's not a reason to disregard the terms — it's context for proportionate decision-making, and account termination is a genuinely bad outcome if the account matters to you.

Step 3: Check for a sanctioned route

The step people skip, and it's often the answer.

Amazon operates a Product Advertising API as part of its Associates programme, with its own terms and eligibility requirements. Many large sites have equivalents:

• Retailers with affiliate or partner APIs
• Marketplaces with seller-facing APIs
• Job boards with syndication feeds
• Public bodies with open data portals

Why an API is better when one fits: it's explicitly permitted, it's structured and stable, it won't break when the site redesigns, it's rate-limited in a documented way, and nobody blocks you for using it as intended.

Why it often doesn't fit: APIs expose a subset of what's on the page, eligibility requirements can be substantial, and rate limits may be lower than you need.

Check first anyway. Ten minutes to discover a free, sanctioned route is ten minutes well spent, and discovering it after building a scraper is annoying.

Applying this to a real decision

Say you want to track prices on twenty products you resell.

Read robots.txt. Check whether product detail pages are disallowed for User-agent: *. Note that search and category listing paths are commonly restricted even where detail pages aren't.

Read the Conditions of Use. Find the automated-access clause. Note whether you have an account — if you do, you've clicked through.

Check for an API. For Amazon specifically, the Product Advertising API exists but has Associates eligibility requirements that not everyone meets.

Now assess honestly:

The honest read: there's a large practical gap between structured note-taking on pages you're browsing and running an automated crawler. Terms often don't distinguish them; risk exposure does, considerably.

Where a browser extension sits. ScrapeMaster runs in your browser on pages you already have open, at the pace you navigate. It doesn't crawl autonomously, doesn't rotate proxies or fingerprints, and doesn't bypass logins or CAPTCHAs. That places it at the low end of the risk spectrum structurally — but it doesn't override terms you accepted. If you're signed in and the terms prohibit automated extraction, that's a contract you're party to regardless of which tool you use. We'd rather say that plainly.

Reading terms on any site: a checklist

1. /robots.txt — find the User-agent: * block, check the paths you care about
2. Footer → Terms — search for the automated-access keywords
3. Note how you're bound — do you have an account? Did you click through?
4. Look for an API or partner programme
5. Check the privacy policy if you'll touch data about people
6. Screenshot or PDF what you read, with the date — terms change, and knowing what they said when you decided is useful
7. Write down your reasoning in three sentences

Step 6 is worth the fifteen seconds. Convert: Web to PDF captures the terms page as a dated PDF with selectable text and working cross-reference links, entirely on your machine. Terms pages get rewritten and rarely keep public version histories — six months from now, the version you actually read may not exist anywhere.

Step 7 matters more than it sounds. A short contemporaneous note — what you read, what you concluded, why — is far more useful than reconstructing your reasoning later.

Frequently asked questions

Where do I find a site's rules about automated access?

Three places: /robots.txt at the site root for path-level directives, the terms of service or conditions of use (linked in the footer) for the contractual position, and any developer or API terms for the sanctioned route. Search the terms for "robot", "scraper", "crawl", "automated", and "data mining" to find the relevant clause in seconds.

Does Amazon's Conditions of Use prohibit scraping?

Amazon's Conditions of Use include a licence-and-access provision restricting data mining, robots, and similar data gathering and extraction tools absent express written consent — the standard shape for large commercial sites. Amazon also operates a Product Advertising API through its Associates programme as a sanctioned route, with its own eligibility requirements. Read the current documents directly; they change.

Is breaking a site's terms of service a crime?

Generally no — it's typically a breach of contract issue rather than a criminal one. US courts narrowed the Computer Fraud and Abuse Act's reach in Van Buren, focusing it on circumventing technical access barriers rather than violating usage policies. Realistic consequences are rate limiting, blocking, and account termination, with legal action reserved for large-scale or commercially damaging cases.

How does robots.txt actually work?

It lists directives grouped by user-agent. Only the most specific matching block applies — a client matching a named block ignores the wildcard User-agent: * block entirely rather than combining them. Disallow: lines name path prefixes not to fetch, Allow: carves out exceptions, and Crawl-delay: requests a gap between requests.

Does having an account change whether I can extract data?

Substantially. Creating an account almost always means accepting terms via clickwrap, which courts routinely enforce — unlike footer-linked browsewrap terms, where assent is often not found. Being signed in also means the data isn't public, so the public-data reasoning from cases like hiQ v. LinkedIn doesn't apply.

Should I use the API instead of extracting from pages?

Where an API exists and serves your use case, yes — it's explicitly permitted, structured, stable across redesigns, and documented. The limitations are that APIs typically expose less than the page does, eligibility requirements can be significant, and rate limits may be lower than you need. Check before building anything else.

How do I keep a record of what the terms said when I read them?

Convert the terms page to PDF with the date and URL stamped on it, and keep it. Terms pages are edited in place and rarely maintain public version histories, so the version you relied on may become unrecoverable. A local browser-based conversion keeps the record on your machine and works on pages behind a login.

Bottom line

The documents governing whether you can extract from a site are public, short, and almost never read before the argument starts. /robots.txt, the terms of service, and the developer docs — ten minutes, total.

Read the User-agent: * block, search the terms for the automated-access keywords, notice whether you clicked through anything, and check whether a sanctioned API exists. That's the whole method, and it transfers to every site.

Then be honest with yourself about which activity you're actually proposing. Structured note-taking on pages you're browsing and an automated crawler enumerating a catalogue are treated identically by most terms and very differently by everyone's risk exposure — including yours.

Capture what you read, note why you decided what you decided, and move on.

ScrapeMaster works on pages you already have open, at browsing pace, without bypassing access controls — and never makes your use compliant on its own. For the broader legal picture, see is web scraping legal and are web crawlers legal.

And for when you're done reading legal documents: CineMan AI picks something to watch.