How to use the Remove Duplicate Lines
- Paste your list into the left box, one item per line.
- Choose whether capital letters and stray spaces should matter, and whether to drop empty lines.
- Read the cleaned list on the right, with a count of how many duplicates were removed.
- Copy the result or download it as a text file.
Features
- Keeps the first occurrence of each line, in its original order
- Optional case-insensitive matching
- Trims invisible spaces that make identical lines look different
- Removes empty lines in the same pass
- Shows the most repeated lines and how often each appeared
- Runs in your browser, so private lists never leave your device
Where duplicate lines come from
Duplicates creep in whenever lists are combined. Merging two exports of a mailing list, pasting keyword ideas from several tools, collecting tags from many posts or copying a spreadsheet column that already had repeats all leave the same entry more than once. Removing them by eye is slow and error-prone once a list passes a few dozen lines.
How the tool decides two lines match
By default, two lines are duplicates only when they are exactly the same after trimming spaces from both ends. The options let you loosen that:
| Lines | Default | Ignore case | Trim off |
|---|---|---|---|
| “Apple” and “apple” | Kept both | Duplicate | Kept both |
| “apple” and “apple ” | Duplicate | Duplicate | Kept both |
| “apple” and “apples” | Kept both | Kept both | Kept both |
Trailing spaces are the most common reason a “duplicate” survives in spreadsheets and databases, because you cannot see them. Leave trimming on unless leading indentation is meaningful, as in code. Ignoring case is right for email addresses, domain names and tags, but wrong for things like passwords or case-sensitive product codes.
Why original order matters
Many deduplication methods sort the list first, which destroys any order you cared about: priority, date, ranking or the sequence of steps. This tool walks through the list from top to bottom and keeps the first time it sees each line, so the result reads in the same order as your input, just shorter. If you also want the list alphabetized, run the result through Sort Lines afterwards.
When case is ignored, the first spelling wins. If your list contains “iPhone” and later “IPHONE”, the output keeps “iPhone”.
Seeing what was repeated
The table under the results lists the lines that appeared most often and how many times. That is useful on its own: it shows the most common error in a log file, the most frequent tag in an export, or which address was entered several times in a sign-up sheet.
Doing the same on the command line
If you work in a terminal, sort -u file.txt removes duplicates but sorts the lines, and awk '!seen[$0]++' file.txt keeps the original order, which is what this tool does. The browser version is handier for one-off lists, especially on a phone or a locked-down work computer.
Tips
- Extracting addresses from a messy document? The Email Extractor finds and deduplicates them for you.
- If lines differ only by extra internal spaces, clean them first with Remove Extra Spaces.
- To compare two lists and see what changed, use Text Compare.
Frequently asked questions
Does it keep the first or the last occurrence?
The first. Each line is kept the first time it appears and later copies are dropped, so the list stays in its original order.
How do I remove duplicates regardless of capital letters?
Tick “Ignore upper/lower case”. Lines such as “Apple” and “APPLE” are then treated as the same, and the first spelling is kept.
Why are two lines that look identical both kept?
They probably differ in an invisible way, such as a trailing space, a tab or a non-breaking space. Keep trimming on, or clean the text with Remove Extra Spaces first.
How many lines can I process?
There is no fixed limit. Lists of tens of thousands of lines are handled almost instantly; very large files depend on your device’s memory.
Is my list uploaded anywhere?
No. Everything runs in your browser, so email lists and other private data stay on your device.