Remove duplicate lines in order
First-occurrence wins, original order preserved. Optional case-insensitive matching and whitespace trimming.
apple banana cherry date APPLE
How to dedupe lines
This tool removes duplicate lines while preserving the original order of first occurrences. Toggle case-sensitivity, whitespace trimming, and empty-line handling depending on your data.
- 01
Paste your list
One item per line. Lists, emails, hostnames, log lines — anything.
Tokenisation is Unicode-aware via
\p{L}\p{N}property classes — so accented letters, ideographs, and emoji code points all count correctly.any listUTF-8Unicodecode pointsliveinput · textraw · UTF-8apple banana apple
loaded · UTF-8UTF-8 · ready - 02
Set options
Case-sensitive: APPLE ≠ apple. Trim: leading/trailing whitespace ignored. Keep empty: blank lines retained.
casetrimemptyoptionsconfiguredcase-insens, trim
set - 03
First occurrence wins
Order is preserved — the first occurrence of each unique value stays in place. Use Sort Lines after if you want a sorted unique list.
Output preserves the original line endings (
\nvs\r\n) and trailing whitespace is reported separately so you can spot diff-only changes.stableline endingswhitespacediff-cleanreproducibleoutput · counteddeduped · readyapple banana
counted · livecounted - 04
Copy or download
One-click copy or download as plain text.
Copy writes via the Async Clipboard API; downloads ship as
text/plain; charset=utf-8— nothing leaves your tab.copyclipboardtext/plainblobclient-sideexport · copiedready · done✓ Copied
shipped · ready to shipsuccess
Text-processing fundamentals worth knowing
Tokenise with Unicode classes
Use \p{L}\p{N} with the u flag so accents, ideographs, and emoji count properly.
Normalise line endings
Convert \r\n to \n before diffing or hashing — Windows-vs-Unix endings produce phantom changes.
Mind invisible characters
Zero-width joiners, BOMs, and non-breaking spaces look identical to ASCII space. Strip them before comparison or storage.
Stream don’t slurp
For multi-megabyte text, iterate line-by-line rather than loading everything to memory — keeps the tab responsive.
Choose collation deliberately
Default JS sort is code-point order, not human alphabetical. Use Intl.Collator for locale-aware sorting.
Re-run after normalisation
NFC vs NFD changes byte length but not visible glyphs. String.prototype.normalize() before counts and comparisons.
When to dedupe
Anywhere unique values matter.
Email lists
Clean a mailing list before importing to ESP. Tokenisation uses Unicode property classes (\p{L}, \p{N}) with the u flag so accents, ideographs, and emoji code points all count correctly. Surrogate-pair-aware counts mean 🚀 weighs 1, not 2.
Unicode-correctLog triage
Reduce repeated error lines to unique signatures. Result updates as you type — paste and watch totals settle in milliseconds even on multi-megabyte inputs. Heavy work runs through a memoised reducer so the editor never janks.
diff-cleanURL lists
Remove duplicate URLs from scraper output. Output normalises line endings to LF for clean diffs across platforms. Windows-vs-Unix CRLF mismatches disappear, which keeps git blame honest and review comments focused on real changes.
instant countsTest fixtures
Ensure fixture data has unique IDs. Code-point counts are stable across NFC/NFD normalisation. Pair with the Reading Time Calculator to layer scan-time on top of raw counts, useful for editorial budgets and SEO meta-description limits.
platform-awareBookmarks
Clean up duplicates after a CSV export. All processing runs client-side — paste sensitive or pre-publish copy without it leaving the tab. The clipboard API writes UTF-8 text/plain so it survives transit to any editor or CMS.
code-point-accurateConfigs
Deduplicate hosts in /etc/hosts or allowlists. Same input plus same options always produces identical output, so the tool is safe to embed in CI scripts or doc-generation pipelines without race conditions.
losslessFrequently asked questions
Something missing?
Tell us what you need. We ship the top requests every two weeks.