Remove Duplicate Lines

Paste a list and the repeats disappear as you type. No run button, nothing to wait for, and nothing you paste leaves your browser.

The part worth reading is what this tool counts as the same line, because that is where dedupers quietly disagree with each other and it is where your list comes out right or comes out wrong. Take these six lines: alice@example.com, then Alice@Example.com, then alice@example.com with a trailing space, then bob@example.com, then a blank line, then bob@example.com again. With all three switches off, which is where you land, you get five lines back. Tick Trim and it is four. Add Case-insensitive and it is three. Add Also remove empty lines and you are down to two. Same paste, four different answers, and which one you want depends on what the list is for.

There are two modes and only the first is common. Remove duplicates (keep one) collapses each group down to its first copy and leaves your order alone. Keep only unique lines does the job a plain deduper cannot do at all: it throws the whole group out and hands back the entries that appeared exactly once. Paste two exports together and you want the rows that are in one file and not the other, that is this mode.

1. Paste your text into the top box, one item per line.
2. Pick a mode. Both mode buttons stay dead until there is something in the box, so paste first and choose second. Keep-one is what you get by default.
3. Turn on the switches you need. All three start off, which means a strict character for character match. Case-insensitive folds Apple and apple together, Trim ignores spaces and tabs at either end of a line, and Also remove empty lines clears the blanks.
4. Read the counts. Original, Removed and Remaining update on every keystroke, so you can see what a switch cost you before you copy anything.
5. Copy result, or Clear to empty the box. Clear leaves your mode and your switches exactly where they were.

Every switch changes how two lines are compared, and none of them changes the text that comes out.

Put report with a trailing space on the first line and plain report on the second, turn on Trim, and you get one line back. That line is still report with the space on it. The first copy is the one that survives and the trim only ever happened inside the comparison. Capitals behave the same way, REPORT above report with Case-insensitive on returns REPORT rather than a lowercased version.

I would not want it any other way, since a tool that quietly rewrote your data while it deleted rows would be much worse than one that leaves a stray space in. It does mean deduping is not cleaning, so if the whitespace itself is the problem, run text cleaner after this rather than instead of it.

This tool folds accents together but leaves lookalike letters alone, and the split is not the one you would guess. I tried twelve pairs that look identical or near enough on screen and read the result off the tool. Four of them matched and the other eight came back as two lines.

Matched, one line out:

• café written with a single é against café written as a plain e plus a separate accent mark, which is four characters against five
• the Kelvin sign at U+212A against an ordinary capital K
• the Angstrom sign against Å
• the Ohm sign against a Greek capital omega

Did not match, two lines out:

• full width Ａ against A
• the fi ligature against f followed by i
• Jane Doe joined by a no-break space against the same name with an ordinary space
• invoice with a zero width space stuck on the end
• invoice with a soft hyphen stuck on the end
• Cyrillic а against Latin a, and Greek ο against Latin o
• a non-breaking hyphen against a normal hyphen

The accent case is the one that helps you, and it is on by default because a name exported from a Mac address book really is the same name as the one exported from a database even though the two spellings are different lengths. The eight that stay apart are the honest limit. Cyrillic а is a different letter to a computer and no amount of trimming will change that, so if you have run out of switches and the duplicates are still sitting there, you are looking at an invisible character or a lookalike. Paste one of the stubborn pairs into text cleaner and it will name what it found.

The accent fold reaches further than accents, which is worth one warning. 486 single characters in the basic Unicode range turn into a different single character before this tool compares anything. Type Kg using the Kelvin sign, then Kg using a plain K, and you get one line back that still holds the Kelvin sign, because the first copy is the survivor. It looks like an ordinary K on screen, and the next system you paste it into will read it as a different character.

A blank line is a line like any other, so every blank in your paste is a duplicate of every other blank. Keep-one leaves exactly one of them in the entire output.

I pasted a twelve paragraph draft, 24 lines with a blank between each paragraph, and got 13 lines back. Every paragraph survived along with exactly one of the twelve blank lines, which is now sitting after paragraph one with the other eleven paragraphs jammed together underneath it. In Keep only unique lines the same paste loses every blank, because a blank that occurs 12 times is not a singleton.

So this is a tool for lists. Emails, SKUs, IDs, log lines, one item per row, and there is no setting that makes a document safe to run through it.

One related trap: a line of three spaces is not empty. Also remove empty lines on its own clears only lines that are truly nothing at all, and a five line paste with a spaces-only row and a tabs-only row still came back with four lines. Add Trim and it dropped to two. That combination is the fix for text pulled out of a PDF, which loves to leave rows that look blank and are not.

If you already have a terminal open, be honest about whether you need a web page for this. Take six lines: zebra, alpha, zebra, mango, alpha, kiwi.

• sort -u returns alpha, kiwi, mango, zebra. The right set, and your order is gone.
• sort | uniq -u returns kiwi, mango. The right singletons, alphabetized, because uniq needs sorted input by design and sorting is what costs you the order.
• awk '!seen[$0]++' returns zebra, alpha, mango, kiwi. Order kept, but it only does the keep-one job.
• This tool on keep-one returns zebra, alpha, mango, kiwi. On Keep only unique lines it returns mango, kiwi, in the order they arrived.

That last row is the narrow, real case for using this instead: you want the lines that occurred once AND you want them in their original order. If order does not matter to you, sort -u is faster than opening a browser. I would use it.

There is no cap in the tool, so I went looking for the ceiling instead of guessing at it. 40,000 lines and 857,779 characters came back in about 6 ms. 100,000 lines and 2,177,779 characters took 16 ms. A million lines and 22,777,779 characters, which is a 22 MB paste, took 314 ms as the median of three runs. Keep only unique lines costs roughly 1.7 times as much as keep-one, since it has to count every line before it can decide about any of them, 181 ms against 106 ms on half a million lines.

There is a catch, though, and it is why I would not paste 22 MB in on purpose. The whole list is redone on every keystroke. Add one more character to a million line paste and you wait 354 ms for it, every time. At 100,000 lines that same cost is 16 ms and you will never notice. Paste, copy and move on and the size does not matter. It is only editing inside a giant paste that drags.