Remove Duplicate Lines
Private: nothing leaves your browser
Waiting for text
Paste a list and get it back with the repeats gone and the order untouched. The first copy of each line stays where it was. You can also ignore case, ignore stray spaces, and drop blank lines while you are at it.
About removing duplicate lines
Duplicates creep into a list every time two exports get merged, a spreadsheet column is copied twice, or a log is filtered by hand. Sorting first and eyeballing the result works for twenty lines and stops working at two hundred. This reads the whole list in one pass, keeps the first copy of every line and drops the rest, so a list of ten thousand email addresses comes back clean in the time it takes to paste it.
Order is preserved deliberately. Most tools that remove duplicates sort the list as a side effect, which destroys a priority order or a chronological order that took work to build. If you do want the result sorted, run it through Sort Lines afterwards, and you get to decide rather than having it decided for you.
The comparison options matter more than they look. Ignore case treats "Mumbai" and "mumbai" as the same entry, which is what you want for names and email addresses and not what you want for case sensitive identifiers. Ignore leading and trailing spaces catches the very common case where a paste has picked up a stray space, so two lines look identical on screen but are not. In both cases the line that is kept is the original, exactly as you typed it: only the comparison changes, never your text.
How to remove duplicate lines
- Paste your list into the left box, one item per line.
- The unique lines appear on the right straight away, and the status line tells you how many were removed.
- Tick Ignore case if "Delhi" and "delhi" should count as the same entry.
- Tick Ignore leading and trailing spaces if your paste may have picked up stray whitespace.
- Tick Remove blank lines to close up the gaps at the same time.
- Press Copy for the result, or Result to input to keep working on it.
Common questions about removing duplicate lines
Which copy of a repeated line is kept?
Does it sort my list?
Do duplicates have to be next to each other?
uniq command, which only collapses repeats that are already adjacent and is why it is usually paired with sort.
Why are two lines that look identical both kept?
How many lines can it handle?
Is it safe to paste a list of customer emails in here?
Next