TL;DR
Exact-match deduplication catches almost nothing in a scraped lead list, because duplicates rarely look identical. The same company arrives as Acme Corp, Acme Corporation, and ACME Corp. — three rows, three spellings, one company.
Deduplicate on the domain, not the name. A company has many names and usually one website. Normalizing to a bare domain (acme.com) and grouping on that catches the overwhelming majority of company-level duplicates in one pass, with no fuzzy matching and no tooling beyond a spreadsheet.
Then apply, in order:
- Domain match — the primary key. Catches most of it.
- Normalized name match — for rows with no website, or where the domain differs (regional sites, redirects).
- Fuzzy name match — for the remainder, reviewed by a human, never auto-merged.
- Person-level dedup within a company — the same individual on both a team page and a press release.
The cheapest place to fix this is before import. Cleaning a CSV is a spreadsheet task; cleaning a CRM after a bad import means untangling merged activity histories, and it's an order of magnitude more work.
ScrapeMaster helps at the front of this by producing consistent columns across sources — you can name the extracted columns the same way on every site, so the URL field is always called the same thing and the domain formula works everywhere without per-source rework.
Why duplicates are guaranteed, not a mistake
If you collect from more than one source, duplicates aren't a sign you did it wrong. They're arithmetic. A directory, an association member list, and a conference exhibitor list will overlap, because active companies appear in all three.
Where duplicates come from, in rough order of frequency:
Name variants. Legal name versus trading name versus how the marketing team writes it. Acme Corporation, Acme Corp, ACME, Acme Group Ltd, Acme (UK) Limited. All the same entity, none matching on a string comparison.
URL variants. acme.com, www.acme.com, https://acme.com/, acme.com/en/, acme.com?utm_source=directory. Five strings, one domain.
Subsidiary and regional entries. Acme GmbH and Acme Inc. may be genuinely distinct legal entities you want separately, or the same commercial relationship you want merged. This is a business decision, not a data one, and you should decide it before you start rather than during.
Re-runs. Scraping the same source twice — a week later, or after a filter change — reproduces rows you already have.
Multiple contacts at one company. Not a duplicate at the company level, and often exactly what you wanted. But if your pipeline treats each row as an account, five contacts at Acme become five accounts.
The method
Step 1: Normalize the domain
This is the highest-leverage step and it's a single formula.
Take whatever URL column you have and reduce it to a bare domain: strip the protocol, strip www., strip everything after the first /, strip query parameters, lowercase it.
In Excel or Google Sheets, given a URL in A2:
=LOWER(
IFERROR(
REGEXEXTRACT(A2, "^(?:https?://)?(?:www\.)?([^/?#]+)"),
""
)
)
In Sheets that works as written. In Excel, use TEXTAFTER/TEXTBEFORE or Power Query's URL parsing if REGEXEXTRACT isn't available in your version.
Now https://www.Acme.com/en/about?utm_source=x and acme.com both become acme.com, and a plain COUNTIF finds the duplicates:
=COUNTIF($B$2:$B$5000, B2) > 1
Sort by that column and every domain-level duplicate is grouped together.
Watch for two traps. Shared hosting domains — several small businesses on the same platform subdomain — will collapse unrelated companies. And multi-brand groups legitimately run several domains. Scan the high-count groups by eye before merging; a domain appearing 40 times is a signal to look, not to merge.
Step 2: Normalize the company name
For rows with no URL, and as a cross-check on step 1.
Apply consistently:
- Lowercase everything.
- Strip legal suffixes:
inc,llc,ltd,limited,corp,corporation,gmbh,ag,bv,sa,srl,plc,pty,co. - Strip punctuation and extra whitespace.
- Strip leading articles:
the.
Acme Corporation → acme. ACME Corp. → acme. The Acme Group Ltd → acme group.
Nested SUBSTITUTE calls do this in a spreadsheet. It's ugly and it works:
=TRIM(LOWER(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(
A2, ",", ""), ".", ""), " Inc", ""), " Ltd", "")))
Extend the chain for whichever suffixes appear in your data — check what's actually there rather than guessing.
Now COUNTIF on the normalized name catches the pairs the domain missed.
Step 3: Fuzzy match the remainder
After steps 1 and 2 you'll have a small tail: typos, transliterations, abbreviations (Intl vs International), and genuinely ambiguous cases.
Two practical options:
Sort and eyeball. Sort by normalized name and scroll. Similar names sort adjacently, and on a list of a few thousand this takes ten minutes and is more accurate than any algorithm, because you can tell that Acme Systems and Acme Solutions are different companies and a string-distance metric can't.
Length-and-prefix grouping. Compare the first several characters of the normalized name plus its length. Crude, but it surfaces near-matches without needing a Levenshtein implementation.
Never auto-merge on fuzzy match. Flag for review. The cost of a false merge — two real companies collapsed into one record, with mixed activity history — is much higher than the cost of one duplicate slipping through.
Step 4: Deduplicate people within a company
Different problem, different key. Once companies are resolved, the same person may appear twice from different sources.
Key on normalized name + domain, not on email — the same person often has a role-based address on one source and a direct address on another, and matching on email would treat them as two people.
Normalize the person's name the same way: lowercase, strip punctuation, strip titles (Dr, Prof, Mr, Ms), and handle the Last, First versus First Last inconsistency that different sources produce.
Step 5: Decide which row survives
For each duplicate group, pick a winner:
- Most complete row. Count non-empty fields and keep the highest.
- Most recent. If you tracked a collection date, prefer the newer.
- Most authoritative source. The company's own website beats a third-party directory.
Then merge the fields rather than discarding rows outright — a duplicate may carry a phone number the winner lacks. Combining before deleting salvages real data.
Prevention beats cleanup
Most of this work disappears if you set up the collection properly.
Use consistent column names across sources. When you extract, rename columns to a fixed schema — company_name, website, contact_name, contact_title, phone, email, source, collected_date — on every source. ScrapeMaster saves your column setup per domain, so revisiting a site re-applies the same configuration and you don't drift.
Always capture the URL. It's your primary key. A row without a website is a row you'll deduplicate by hand.
Add a source column at collection time. When you find a duplicate later, knowing which source it came from tells you which sources overlap — and lets you stop scraping a redundant one.
Add a collected-date column. Makes re-runs identifiable and lets you prefer fresher data on merge.
Normalize the domain at extraction, not later. If you can capture the bare domain during collection, your dedup key exists from the start.
Keep a suppression list. Companies you've disqualified, and individuals who asked not to be contacted. Check every new list against it before import. Re-adding someone who opted out is worse than a duplicate — it's a compliance failure and, under GDPR and CASL, a reportable one.
Doing it with more than a spreadsheet
If the list runs past tens of thousands of rows, the spreadsheet approach gets slow.
Power Query in Excel handles the normalization and grouping steps repeatably, and re-runs on refresh, which matters if you re-scrape periodically.
Python and pandas — df['domain'].duplicated() plus rapidfuzz for the fuzzy tail. An hour to set up, then reusable forever. The right answer if this is a recurring pipeline.
Your CRM's own dedup rules — most have them. Configure them before the import rather than running a merge afterwards, and set them to match on domain rather than name.
For a one-off list of a few thousand, the spreadsheet method is genuinely faster than setting any of these up.
Preserving the audit trail
One habit worth adopting: before you deduplicate, save the raw extracted list. Export the pre-dedup CSV and keep it.
If a merge turns out to be wrong — two real companies collapsed — the original is the only way back. And if anyone asks where a contact came from, the raw file with its source column is the answer.
For a fixed, readable snapshot that doesn't depend on spreadsheet software or locale settings mangling dates, convert the raw CSV with Convert: Anything to PDF — it renders as a formatted table with header styling. Set landscape and A3 or Ledger for wide exports.
Frequently asked questions
How do I find duplicate leads by company in a scraped list?
Normalize each row's URL to a bare domain — lowercase, no protocol, no www., no path or query — and use COUNTIF on that column. Domain matching catches most company-level duplicates in one pass, because a company has many name spellings and usually one website.
Why doesn't exact name matching find duplicates?
Because duplicates almost never share a spelling. The same company appears as Acme Corp, Acme Corporation, and ACME Corp. across sources, and an exact string comparison treats those as three companies. Normalizing — lowercase, strip legal suffixes and punctuation — fixes most of it.
Should I deduplicate before or after importing to my CRM?
Before, always. Cleaning a CSV is a spreadsheet task. Cleaning a CRM after a bad import means untangling merged activity histories, duplicate accounts with split notes, and possibly duplicate outreach to the same person — an order of magnitude more work.
How do I handle subsidiaries and regional entities?
Decide the rule before you start. Acme GmbH and Acme Inc. may be separate legal entities you want tracked separately, or one commercial relationship you want merged. It's a business decision about how you sell, not a data-quality question, and deciding it mid-cleanup produces an inconsistent list.
Is fuzzy matching safe to run automatically?
No. Flag fuzzy matches for human review instead. A false merge collapses two real companies into one record with mixed history, which is considerably more damaging and harder to detect than one duplicate that slipped through.
What's the best key for deduplicating people rather than companies?
Normalized person name plus normalized domain. Don't key on email — the same individual often appears with a role-based address from one source and a direct address from another, and email matching would treat them as two people.
How do I stop duplicates happening in the first place?
Use a fixed column schema across every source, always capture the website URL, and add source and collected-date columns at extraction time. ScrapeMaster saves your column configuration per domain, so returning to a site re-applies the same setup and your schema doesn't drift between runs.
Should I keep the original list before deduplication?
Yes. Export the raw extracted CSV before you merge anything. It's your only route back from a wrong merge, and with a source column it's also how you answer "where did this contact come from" later.
Bottom line
Company-level duplicates are inevitable in multi-source collection and mostly solvable with one idea: deduplicate on the domain, because a company has many names and one website. Normalize the URL, COUNTIF it, and the bulk of the problem resolves in a few minutes.
Handle name variants for the rows without URLs, eyeball the fuzzy tail rather than automating it, merge fields before deleting rows, and keep the raw list. Then fix it upstream — consistent columns, always capture the URL, always record the source and date — so the next list needs less of this.
ScrapeMaster is free with no row limits, saves your column setup per domain so your schema stays consistent across runs, and exports to CSV, XLSX, JSON, or clipboard. Extracted data stays local in your browser.
After the list is clean: CineMan AI puts IMDb and Rotten Tomatoes ratings directly on Netflix, Prime Video, and Disney+.