Site riiiightt.com

Scripted Solutions to Remove Duplicate Lines: awk, sed, Python & PowerShell

Written by

admin-dfv33

in

Remove Duplicate Lines Quickly: 7 Easy Methods for Windows, macOS & Linux

Below are seven practical, fast ways to remove duplicate lines from text files on Windows, macOS, and Linux. Each method shows the command or steps and notes about order preservation, ease, and when to use it.

1) sort + uniq (Linux/macOS)

  • Command: 
    bash
    sort file.txt | uniq > out.txt
  • Notes: Removes duplicates but sorts lines (order lost). Fast for large files.

2) awk preserving first occurrence (Linux/macOS, Windows via WSL or Git Bash)

  • Command: 
    bash
    awk ’!seen[\(0]++'</span><span> file.txt </span><span class="token" style="color: rgb(57, 58, 52);">></span><span> out.txt </span></code></div></div></pre> </li> <li>Notes: Preserves original order, efficient, no external tools required.</li> </ul> <h3>3) perl one-liner (cross-platform)</h3> <ul> <li>Command: <pre><div class="XG2rBS5V967VhGTCEN1k"><div class="nHykNMmtaaTJMjgzStID"><div class="HsT0RHFbNELC00WicOi8"><i><svg width="16" height="16" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill="currentColor" fill-rule="evenodd" clip-rule="evenodd" d="M15.434 7.51c.137.137.212.311.212.49a.694.694 0 0 1-.212.5l-3.54 3.5a.893.893 0 0 1-.277.18 1.024 1.024 0 0 1-.684.038.945.945 0 0 1-.302-.148.787.787 0 0 1-.213-.234.652.652 0 0 1-.045-.58.74.74 0 0 1 .175-.256l3.045-3-3.045-3a.69.69 0 0 1-.22-.55.723.723 0 0 1 .303-.52 1 1 0 0 1 .648-.186.962.962 0 0 1 .614.256l3.541 3.51Zm-12.281 0A.695.695 0 0 0 2.94 8a.694.694 0 0 0 .213.5l3.54 3.5a.893.893 0 0 0 .277.18 1.024 1.024 0 0 0 .684.038.945.945 0 0 0 .302-.148.788.788 0 0 0 .213-.234.651.651 0 0 0 .045-.58.74.74 0 0 0-.175-.256L4.994 8l3.045-3a.69.69 0 0 0 .22-.55.723.723 0 0 0-.303-.52 1 1 0 0 0-.648-.186.962.962 0 0 0-.615.256l-3.54 3.51Z"></path></svg></i><p class="li3asHIMe05JPmtJCytG wZ4JdaHxSAhGy1HoNVja cPy9QU4brI7VQXFNPEvF">bash</p></div><div class="CF2lgtGWtYUYmTULoX44"><button type="button" class="st68fcLUUT0dNcuLLB2_ ffON2NH02oMAcqyoh2UU MQCbz04ET5EljRmK3YpQ CPXAhl7VTkj2dHDyAYAf" data-copycode="true" role="button" aria-label="Copy Code"><svg viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill="currentColor" fill-rule="evenodd" clip-rule="evenodd" d="M9.975 1h.09a3.2 3.2 0 0 1 3.202 3.201v1.924a.754.754 0 0 1-.017.16l1.23 1.353A2 2 0 0 1 15 8.983V14a2 2 0 0 1-2 2H8a2 2 0 0 1-1.733-1H4.183a3.201 3.201 0 0 1-3.2-3.201V4.201a3.2 3.2 0 0 1 3.04-3.197A1.25 1.25 0 0 1 5.25 0h3.5c.604 0 1.109.43 1.225 1ZM4.249 2.5h-.066a1.7 1.7 0 0 0-1.7 1.701v7.598c0 .94.761 1.701 1.7 1.701H6V7a2 2 0 0 1 2-2h3.197c.195 0 .387.028.57.083v-.882A1.7 1.7 0 0 0 10.066 2.5H9.75c-.228.304-.591.5-1 .5h-3.5c-.41 0-.772-.196-1-.5ZM5 1.75v-.5A.25.25 0 0 1 5.25 1h3.5a.25.25 0 0 1 .25.25v.5a.25.25 0 0 1-.25.25h-3.5A.25.25 0 0 1 5 1.75ZM7.5 7a.5.5 0 0 1 .5-.5h3V9a1 1 0 0 0 1 1h1.5v4a.5.5 0 0 1-.5.5H8a.5.5 0 0 1-.5-.5V7Zm6 2v-.017a.5.5 0 0 0-.13-.336L12 7.14V9h1.5Z"></path></svg>Copy Code</button><button type="button" class="st68fcLUUT0dNcuLLB2_ WtfzoAXPoZC2mMqcexgL ffON2NH02oMAcqyoh2UU MQCbz04ET5EljRmK3YpQ GnLX_jUB3Jn3idluie7R"><svg fill="none" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path fill="currentColor" fill-rule="evenodd" d="M20.618 4.214a1 1 0 0 1 .168 1.404l-11 14a1 1 0 0 1-1.554.022l-5-6a1 1 0 0 1 1.536-1.28l4.21 5.05L19.213 4.382a1 1 0 0 1 1.404-.168Z" clip-rule="evenodd"></path></svg>Copied</button></div></div><div class="mtDfw7oSa1WexjXyzs9y" style="color: var(--sds-color-text-01); font-family: var(--sds-font-family-monospace); direction: ltr; text-align: left; white-space: pre; word-spacing: normal; word-break: normal; font-size: var(--sds-font-size-label); line-height: 1.2em; tab-size: 4; hyphens: none; padding: var(--sds-space-x02, 8px) var(--sds-space-x04, 16px) var(--sds-space-x04, 16px); margin: 0px; overflow: auto; border: none; background: transparent;"><code class="language-bash" style="color: rgb(57, 58, 52); font-family: Consolas, "Bitstream Vera Sans Mono", "Courier New", Courier, monospace; direction: ltr; text-align: left; white-space: pre; word-spacing: normal; word-break: normal; font-size: 0.9em; line-height: 1.2em; tab-size: 4; hyphens: none;"><span>perl -ne </span><span class="token" style="color: rgb(163, 21, 21);">'print unless \)seen{$}++’ file.txt > out.txt
  • Notes: Preserves order, good for complex line handling and Windows Perl installs.

4) PowerShell (Windows)

  • Command: 
    powershell
    Get-Content file.txt | Select-Object -Unique | Set-Content out.txt
  • Notes: Preserves first occurrence order. Works in PowerShell 3.0+.

5) sort -u (Linux/macOS)

  • Command: 
    bash
    sort -u file.txt -o out.txt
  • Notes: Shortcut that sorts and removes duplicates in-place; order lost.

6) Python (cross-platform, preserves order)

  • Script: 
    python
    seen = set()
with open(‘file.txt’) as fin, open(‘out.txt’,‘w’) as fout:
for line in fin:
        if line not in seen:
            fout.write(line)
            seen.add(line)
  • Notes: Handles large files reasonably; easy to extend (case-insensitive, trimming).

7) Text editor plugins / GUI tools (Windows/macOS)

  • Examples: Notepad++ (TextFX / built-in Remove Duplicate Lines), VS Code extensions, BBEdit
  • Notes: Best for ad-hoc editing, interactive review, and small files.

Additional tips:

  • To remove duplicates ignoring leading/trailing whitespace, trim lines first (awk/perl/Python).
  • For case-insensitive deduplication, normalize case before checking.
  • For very large files where memory is limited, use sorting-based methods or tools that stream and temporary-store keys on disk.

If you want, tell me your OS and whether you need to preserve order or handle case/whitespace and I’ll give the exact command tuned to that.

Comments

Leave a Reply

More posts