Deduplicate Lines
Remove duplicate lines from any list or file with a full unified diff view. See every kept and removed line before committing the change.
How to use the deduplication tool
Paste your content — one item per line. The diff panel shows the result immediately: removed lines in red with a − prefix, kept lines in neutral styling. Adjust the options (case-insensitive, trim whitespace, sort) and the diff updates live. When you're satisfied, click Replace to apply the deduplicated result.
How the diff view is generated
The diff uses a line-level LCS (longest common subsequence) algorithm to produce a unified diff between the original and deduplicated output. Each line is classified as kept, removed, or (for sorted output) reordered. This gives an accurate picture of the transformation without false positives.
Why a diff view matters for deduplication
A silent deduplication — where you just get the cleaned output — is risky when working with code or configuration. A line that “looks like a duplicate” might be an intentional repeat with a different version or flag. The diff view makes every removal explicit and auditable before you accept the change.
Frequently asked questions
How is this different from the editing duplicate remover?
The dev version adds a full unified diff view — every line is shown with a + (kept) or − (removed) prefix, like a git diff. This makes it easy to audit exactly what changed, which matters when deduplicating import lists, config entries, or dependency lists where accidental removal is costly.
What types of content is this tool useful for?
Deduplicating npm package lists, CSS class name collections, import statements, environment variable files, API response arrays pasted as plain text, git file lists, and any structured text where each line is a distinct entity.
Does it preserve the original order?
Yes, by default. The first occurrence of each line is kept in its original position. Enable the Sort option to sort the result alphabetically instead.
How does case-insensitive deduplication work?
With case-insensitive mode on, comparisons are made against the lowercased version of each line. The original casing of the kept entry is preserved in the output — only the comparison is case-insensitive.
Tools you might like
17 tools across three categories — all free, no signup required.
Measure, analyze, and optimize your writing. Count words, estimate reading time, check readability scores, and find which keywords dominate your draft.
Transform and clean text into the format you actually need. Convert case, strip junk whitespace, remove duplicate lines, and generate filler copy in seconds.
Built for code and data work. Convert identifiers between naming conventions, inspect byte counts, strip HTML tags, count lines, and deduplicate with a diff view.