Remove Duplicate Lines — Deduplicate Text Online

Where duplicate lines come from

Log files are the most common source. An error that fires repeatedly writes the same event line hundreds of times in sequence, drowning out the surrounding context. Exported lists from spreadsheet filters often carry duplicate rows when the same record matched multiple filter criteria. Merging two exports of the same dataset — say, a Monday pull and a Friday pull — produces doubles wherever a record appeared in both.

Version control files accumulate duplicates through separate editing sessions. A .gitignore entry added by one developer, then added again by another who did not check, is a typical example. Requirements files and package lists acquire the same pattern. Changelog and to-do files often contain a task that was copied to mark it as in-progress and never had the original removed. Web scraping produces duplicates when a paginated result set overlaps: the first item of page two is frequently the last item of page one.

How to remove duplicate lines

Paste your text into the box. The line counter at the bottom shows how many lines are present before any processing.

Choose your options. Keep first occurrence preserves the line at the position where it appeared earliest in the input. Keep last occurrence updates its position to where it appeared most recently — useful when your data was appended chronologically and the bottom of the file is more current than the top. Ignore case makes "Apple" and "apple" count as the same line. Trim whitespace makes "  hello  " and "hello" count as the same line.

Click Remove Duplicates. The output shows the deduplicated text. The status line reports how many lines were in the input, how many remain, and how many were removed.

A worked example

Input — six lines: apple / banana / Apple / cherry / banana / apple.

Keep first, case-sensitive: four lines remain — apple, banana, Apple, cherry. The second banana and the second apple are removed. Apple survives because it is capitalised differently from apple. Two lines removed.

Keep first, case-insensitive: three lines remain — apple, banana, cherry. Apple is now treated as a duplicate of apple and removed along with the later occurrences. Three lines removed.

Keep last, case-sensitive: four lines remain, but the positions shift. The surviving lines are ordered by their last appearance: Apple, cherry, banana, apple. The relative order among survivors always follows input order — nothing is sorted.

Real-world use cases

Keyword lists for advertising campaigns or SEO planning: uploading duplicate keywords to a platform wastes budget and distorts performance reporting. Running the list through this tool before upload costs ten seconds.

Email and subscriber lists: merging two export files from a mailing platform almost always produces duplicates. A list with duplicates charges per-send twice for the same address and skews open-rate metrics.

Dictionary and word list files: spell-check dictionaries and autocomplete vocabularies grow by appending new words over time. Periodic deduplication keeps lookup performance predictable.

Configuration files: a requirements.txt with the same package twice can cause install conflicts. A .env file with a duplicate key will be read differently by different parsers. Deduplication catches both.

SQL dump normalisation: INSERT statements for the same primary key appearing twice cause errors on import. A quick pass through this tool before running the import finds the problem without loading the file into a database.

Changelog and release notes: a release note entry that was drafted, copied for review, and then copied back often ends with the same line appearing twice. Deduplication catches it before the file is published.

Keep first versus keep last

Keep first preserves the line at its earliest position. Use this for priority-ordered lists, for canonical word lists where the first definition is the authoritative one, and for any sorted input where the top occurrence is the correct one.

Keep last reflects the most recent entry when data was appended in chronological order. A server log where the newest event is at the bottom of the file, or a changelog where the most recent entry comes last, should use keep last so the current state survives when older duplicates are removed.

Neither mode reorders surviving lines. The output is always a subsequence of the input in the original sequence. Duplicates are dropped; everything else stays where it was.

Matching options: case and whitespace

Case-sensitive matching is the default. "Error" and "error" are treated as different lines. Use this when capitalisation carries meaning — proper names, file paths that are case-sensitive on the target filesystem, or any list where casing is part of the identity of the value.

Case-insensitive matching is the right choice for keyword lists, email addresses, and anything typed by multiple people who may not be consistent about capitalisation. The output line uses the casing of the surviving occurrence: if you chose keep first and the first occurrence was "Apple", the output shows "Apple" even though later occurrences were "apple".

Trim whitespace matching treats "  banana  " and "banana" as the same line. The casing and padding of the surviving occurrence are preserved in the output. If you want clean output regardless of which occurrence survives, run the text through the remove extra spaces tool before deduplicating.

Empty lines and blank-only lines

Multiple consecutive empty lines are treated as duplicates of each other by default. Three blank lines in a row become one. This is usually the right behaviour for formatted text where blank lines are used as paragraph separators.

A line containing only spaces is not the same as an empty line unless trim whitespace is enabled. With trim enabled, "    " is treated as "" — an empty line — and is deduplicated with other empty or blank-only lines accordingly.

Your text never leaves your browser

Deduplication is handled by a JavaScript Set running in the current tab. Nothing is uploaded, nothing is logged, and nothing persists after you close the page. For sensitive lists — email addresses, passwords, internal identifiers — this tool is safe to use because the data travels nowhere.

If you need to verify this yourself: open the browser network panel before pasting, then click Remove Duplicates. No outgoing request will appear.

Related tools on this site

If lines contain extra spaces that should be stripped before comparison, run the text through the remove extra spaces tool first, then deduplicate.

If the problem is invisible characters rather than duplicate lines, the invisible character remover handles zero-width spaces and Unicode control characters with a per-character report.

For removing unwanted line breaks from prose before deduplicating, use the remove line breaks tool. For cleaning up symbols from the deduplicated output, the remove special characters tool handles that step.