Remove Duplicate Lines

Deduplicated list — every distinct line, once. The usual job: cleaning an email list, a set of IDs, a log of URLs.

Only lines that repeat — the answer to "what is duplicated in here?", which is a different question and one that a deduplicating tool cannot answer.

Only lines that appear once — the answer to "what is in one list and not the other?" when you have concatenated two lists. Combined with case-insensitivity this catches near-duplicates that a naive comparison misses.

Comparison ignores case and surrounding whitespace by default, because ADA@EXAMPLE.COM and ada@example.com are the same address and treating them as different is how a "deduplicated" mailing list still sends two copies. The output preserves the original text of whichever occurrence was kept.

The first occurrence of each line stays where it was and every later copy is dropped, so the output is your file with holes in it rather than a reordered list. That is the difference between this and the usual command-line habit of sort | uniq, which only works because sorting brings duplicates next to each other and destroys the original order as a side effect. Keeping position matters when the order carries meaning — a log, a playlist, a list of steps — and it is the reason a file that was already sorted comes out looking untouched apart from the removals.