Skip to content

List Deduplicator - Remove Duplicate Lines Online

Deduplicated List

Remove Duplicate Lines

Paste a list with one item per line and the unique items appear in the output straight away, in the order they first appeared. The summary shows how many items you started with, how many are unique and which values repeated. Copy the result or download it as a .txt file.

Example

Pasting apple, banana, apple, cherry, Apple with Split on set to Comma and Case-sensitive off gives apple,banana,cherry and reports apple ×3. With Case-sensitive on, Apple is kept as a separate item and the report says apple ×2.

Options

  • Case-sensitive (on by default) treats Server and server as different items.
  • Trim whitespace (on by default) ignores spaces and tabs at the start and end of each item, the most common reason pasted spreadsheet values fail to match. Blank lines are always dropped.
  • Order keeps first-seen order, or sorts A–Z, or sorts naturally so item2 comes before item10.
  • Split on handles comma-, tab-, semicolon- or pipe-separated values as well as one item per line.

Doing It Elsewhere

On the command line, sort -u file.txt removes duplicates but also sorts, and awk '!seen[$0]++' file.txt keeps the original order. In Excel, Data > Remove Duplicates works on a column but is case-insensitive with no option to change it. Pasting here is quicker for a one-off list and shows you which values repeated.

Related tools: compare two lists to find what is missing from one of them, sort lines numerically, by length or at random, or add a prefix or suffix to every line, for example to wrap each value in quotes.

Related Tools

Frequently Asked Questions

Which copy of a duplicate is kept?
The first one. Later repeats are dropped and the remaining items keep their original order, unless you pick a sort. With Case-sensitive off, the first spelling wins: if Apple appears before apple, the output says Apple.
Is matching case-sensitive?
By default, yes: Apple and apple are two different items. Untick Case-sensitive for email addresses, usernames and most names, where capitalisation is inconsistent but means nothing.
Why are two lines that look identical not treated as duplicates?
Usually invisible characters. Trim whitespace (on by default) handles leading and trailing spaces and tabs, but not a non-breaking space or a zero-width character in the middle of a line. Accented letters typed in two different ways (a precomposed é and an e plus a combining accent) are normalised and do match.
My list is comma-separated. Does that work?
Set Split on to Comma, Tab, Semicolon or Pipe. Items are split on that character as well as on line breaks, and the cleaned output is joined with the same character, so a comma-separated list stays comma-separated.
Can I see which lines were duplicates?
Yes. The summary shows the total, the number of unique items and how many were removed. Expand the duplicates list to see each repeated value with its count, for example apple ×3.
Is my data private?
Yes. The list is processed in your browser and never uploaded, which matters for customer emails and order exports.