List Deduplicator - Remove Duplicate Lines Online
Remove Duplicate Lines
Paste a list with one item per line and the unique items appear in the output straight away, in the order they first appeared. The summary shows how many items you started with, how many are unique and which values repeated. Copy the result or download it as a .txt file.
Example
Pasting apple, banana, apple, cherry, Apple with Split on
set to Comma and Case-sensitive off gives
apple,banana,cherry and reports apple ×3.
With Case-sensitive on, Apple is kept as a separate item
and the report says apple ×2.
Options
- Case-sensitive (on by default) treats
Serverandserveras different items. - Trim whitespace (on by default) ignores spaces and tabs at the start and end of each item, the most common reason pasted spreadsheet values fail to match. Blank lines are always dropped.
- Order keeps first-seen order, or sorts A–Z, or sorts naturally so
item2comes beforeitem10. - Split on handles comma-, tab-, semicolon- or pipe-separated values as well as one item per line.
Doing It Elsewhere
On the command line, sort -u file.txt removes duplicates
but also sorts, and awk '!seen[$0]++' file.txt keeps the
original order. In Excel, Data > Remove Duplicates works on a
column but is case-insensitive with no option to change it. Pasting
here is quicker for a one-off list and shows you which values
repeated.
Related tools: compare two lists to find what is missing from one of them, sort lines numerically, by length or at random, or add a prefix or suffix to every line, for example to wrap each value in quotes.
Related Tools
Markdown Preview
Write Markdown and see the rendered HTML output in real time.
Lorem Ipsum Generator
Generate placeholder text in paragraphs, sentences, or words.
List Diff / Compare
Compare two lists and find items unique to each, shared between both, or missing from one.
Word & Character Counter
Count words, characters, sentences, paragraphs, and estimate reading and speaking time.