HomeText & Developer Tools › Remove Duplicate Lines & Sort

Remove Duplicate Lines & Sort

Paste a list, remove duplicate and empty lines, trim whitespace, sort alphabetically, naturally or by length, reverse or shuffle - then copy the clean result.

       

What this tool does

Lists copied out of spreadsheets, CRMs, log files and keyword tools are rarely clean: the same email shows up three times, half the entries have a trailing space, and there are blank rows where merged cells used to be. This tool treats every line as one item and lets you dedupe, trim, drop blanks and sort in a single pass. Processing happens in your browser, so customer lists and internal URLs never leave your machine.

Order of operations

The steps always run in this order: trim whitespace, remove empty lines, remove duplicates, sort, then reverse. Trimming first means banana and banana are recognized as the same item instead of slipping past the dedupe step. Deduplication keeps the first occurrence of each line and deletes later copies, so with "Keep original order" your list keeps its original sequence minus the repeats. With case-insensitive checked, Apple and apple count as one item and whichever appeared first is kept.

The sort options differ in how they compare text. A to Z uses plain character-code ordering, which puts all capital letters before lowercase (Zebra sorts before apple) unless case-insensitive is on. Natural uses locale-aware comparison with numeric collation, so item9, item10, item100 come out in numeric order instead of the string order item10, item100, item9; it is the right choice for file names, version numbers and SKUs. Shortest first and longest first sort by character count, with ties broken alphabetically. Shuffle randomizes the order (using Math.random, which is fine for picking a raffle order but not for anything security-related).

Worked example and tips

The sample list has 10 lines, including one blank line, banana with padding, and the pair apple / Apple. With the defaults (trim, remove empty, dedupe, case-sensitive) you get 7 lines: the blank is dropped and the second banana and second apple are removed, but Apple survives because the case differs. Check case-insensitive and it drops to 6. Choose Natural sort and the item lines come out as item9, item10, item100.

To dedupe comma-separated values, first run them through the Find and Replace tool to turn commas into line breaks, clean up here, then convert back. To count how many times each line occurs rather than just remove repeats, sort A to Z first so identical lines sit together. Lists of a few hundred thousand lines process in about a second in a modern browser.

Frequently asked questions

Does removing duplicates change the order?

Not by itself. With "Keep original order", the first copy of each line stays where it was and only the later repeats are deleted. Order changes only when you pick a sort option or check Reverse.

How is this different from Excel's Remove Duplicates?

Excel compares cells exactly (it is case-insensitive by default), and it does not trim spaces or drop blank rows for you. Pasting the column here, cleaning it, and pasting back gives the same dedupe plus trimming and sorting in one step.

Why does A to Z put capital letters first?

Plain A to Z compares character codes, and in Unicode every uppercase letter comes before every lowercase letter. Check Case-insensitive, or use Natural sort, for a dictionary-style ordering.

Is the shuffle random enough for a drawing?

It uses a Fisher-Yates shuffle driven by Math.random, which is unbiased and perfectly adequate for picking presentation order or a casual raffle. It is not cryptographic randomness, so do not use it for anything with money on the line.