uniq
Collapse or count adjacent duplicate lines
uniq removes duplicate lines, but only when they're adjacent. It doesn't scan the whole
file for repeats; it just compares each line to the one before it. Unsorted input with
duplicates scattered throughout it will pass straight through uniq unchanged. That's why
uniq almost always shows up right after sort in a pipeline: sort | uniq groups matching
lines together first, so uniq has something adjacent to collapse.
-c prefixes each line with how many times it occurred, which is the basis of the classic
sort | uniq -c | sort -rn frequency-count pipeline. -d prints only lines that had duplicates;
-u prints only lines that didn't. -i folds case before comparing. -f N and -s N skip the
first N fields or characters before comparing, useful when only part of each line should count
toward "duplicate."
Sample files used on this page
Every example below was run against these files. Recreate them to follow along.
wordlist.txt duplicates and mixed case, so -u / -f / -i have something to do
banana
apple
Cherry
apple
date
banana
apple
Elderberry
cherry
access.log 10 requests in the standard combined log format; IP is field 1
203.0.113.5 - - [13/Aug/2026:09:12:01] "GET /index.html HTTP/1.1" 200 512
203.0.113.5 - - [13/Aug/2026:09:12:03] "GET /style.css HTTP/1.1" 200 231
198.51.100.7 - - [13/Aug/2026:09:14:22] "GET /index.html HTTP/1.1" 200 512
198.51.100.7 - - [13/Aug/2026:09:14:25] "GET /missing.html HTTP/1.1" 404 162
203.0.113.5 - - [13/Aug/2026:09:15:47] "GET /index.html HTTP/1.1" 200 512
192.0.2.44 - - [13/Aug/2026:09:16:03] "POST /login HTTP/1.1" 302 0
192.0.2.44 - - [13/Aug/2026:09:16:04] "GET /dashboard HTTP/1.1" 200 4021
198.51.100.7 - - [13/Aug/2026:09:18:51] "GET /index.html HTTP/1.1" 200 512
203.0.113.5 - - [13/Aug/2026:09:19:10] "GET /api/status HTTP/1.1" 500 89
192.0.2.44 - - [13/Aug/2026:09:20:33] "GET /dashboard HTTP/1.1" 200 4021
users.csv a header row plus 6 records
name,age,department
Alice,34,Engineering
Bob,29,Sales
Carol,41,Engineering
Dave,25,Marketing
Erin,38,Sales
Frank,31,Engineering
report.txt 40 numbered lines, each 5 words
line 1 of the report
line 2 of the report
line 3 of the report
…
line 40 of the report
The adjacency rule
uniq only ever compares a line to the one immediately before it.
See uniq fail to remove a non-adjacent duplicate
printf 'a\nb\na\na\n' | uniqThe first and third 'a' aren't next to each other, so only the adjacent pair (lines 3 and 4) collapses. The first 'a' survives untouched.
Show output
a
b
a
Sort first, then uniq actually removes all duplicates
printf 'a\nb\na\na\n' | sort | uniqSorting groups every occurrence of a value together, so uniq's adjacent-only comparison now sees all of them next to each other.
Show output
a
b
Counting and isolating duplicates
Count how many times each line occurs
sort wordlist.txt | uniq -c-c prefixes every output line with its occurrence count, right-aligned.
Show output
1 Cherry
1 Elderberry
3 apple
2 banana
1 cherry
1 date
Show only lines that appeared more than once
sort wordlist.txt | uniq -d-d filters the output to just the lines that had at least one duplicate - the unique-only lines disappear entirely, not just their extra copies.
Show output
apple
banana
Show only lines that appeared exactly once
sort wordlist.txt | uniq -uThe mirror image of -d: only lines with no duplicates at all.
Show output
Cherry
Elderberry
cherry
date
Show every copy of each duplicated line, not just one
sort wordlist.txt | uniq -DUnlike -d, which collapses each duplicate group to a single line, -D prints all the repeated lines in full - useful when you want to review the actual duplicate rows, not just know which values repeat.
Show output
apple
apple
apple
banana
banana
Rank values by how often they occur
sort wordlist.txt | uniq -c | sort -rnThe classic frequency-count pipeline: sort groups matches, uniq -c counts them, a second sort orders by that count. See the sort page for more on the second sort.
Show output
3 apple
2 banana
1 date
1 cherry
1 Elderberry
1 Cherry
Case, fields, and character offsets
Fold case before comparing
sort -f wordlist.txt | uniq -iWithout -i, Cherry and cherry count as different lines. -i compares case-insensitively, so they collapse together - sort -f is needed too, so they're adjacent in the first place.
Show output
apple
banana
Cherry
date
Elderberry
Ignore the first field when comparing
printf '1 apple\n2 apple\n3 banana\n' | uniq -f1-f N skips the first N whitespace-separated fields before comparing - here, ignoring the leading number so the two 'apple' rows count as duplicates despite different first fields.
Show output
1 apple
3 banana
Ignore the first N characters when comparing
printf 'xxapple\nyyapple\nzzbanana\n' | uniq -s2-s N skips the first N characters (not fields) before comparing - useful when duplicates share a fixed-width prefix like a line number or timestamp column that shouldn't count.
Show output
xxapple
zzbanana
Compare only the first N characters
printf 'applesauce\napplepie\nbanana\n' | uniq -w5-w N does the opposite of -s: only the first N characters count toward the comparison, and everything after is ignored. Here, both 'apple...' lines share the same first 5 characters, so the second is treated as a duplicate despite differing after that.
Show output
applesauce
banana
Same character-limit comparison, long-form flag
printf 'applesauce\napplepie\n' | uniq --check-chars=5--check-chars is the exact same flag as -w, spelled out - useful when reading a script that favours long-form flags for readability.
Show output
applesauce
Other input/output forms
Read NUL-separated input instead of newline-separated
printf 'a\0a\0b\0' | uniq -z | tr '\0' '\n'-z (--zero-terminated) works on NUL-terminated records - the safe pairing for filenames from find -print0, same as sort -z.
Show output
a
b
Group matching lines with a blank line between groups
sort wordlist.txt | uniq --group--group prints every line (not just one per group, and not just counts) but inserts a blank line between each distinct value - a middle ground between plain uniq and uniq -c when you want to eyeball the groups themselves.
Show output
Cherry
Elderberry
apple
apple
apple
banana
banana
cherry
date
Finding duplicates in real data
Find which visitors hit the site most, from a real access log
cut -d' ' -f1 access.log | sort | uniq -c | sort -rnExtract just the IP column with cut, then the standard sort | uniq -c | sort -rn frequency pipeline - the same shape works for any column of any log.
Show output
4 203.0.113.5
3 198.51.100.7
3 192.0.2.44
Confirm there are no accidental exact-duplicate rows in a CSV
sort users.csv | uniq -dNo output means no duplicate rows - a quick sanity check to run on any file before treating it as a set of unique records.
File arguments and output forms
Run against a file directly instead of piping
uniq wordlist-sorted.txtuniq takes a filename argument just as readily as stdin - piping from sort is only needed when the file isn't already sorted.
Show output
Cherry
Elderberry
apple
banana
cherry
date
Write the result to a file instead of stdout
uniq wordlist-sorted.txt uniq-out.txtA second filename argument is the output destination - uniq writes there directly and prints nothing to the terminal, no redirection operator needed.
A file with no duplicates passes through unchanged
uniq report.txt | wc -l40 lines in, 40 lines out - uniq only ever removes lines, never adds or reorders them, so a file with no adjacent repeats is untouched.
Show output
40
Combining flags
Count occurrences case-insensitively
sort -f wordlist.txt | uniq -ci-c and -i combine normally: counts folded-case groups instead of exact-case ones. Cherry and cherry now count together as 2.
Show output
3 apple
2 banana
2 Cherry
1 date
1 Elderberry
Count occurrences within a character-limited comparison
printf 'applesauce\napplepie\nbanana\n' | uniq -c -w5-c reports how many lines matched under whatever comparison rule the other flags define - here, only the first 5 characters, so applesauce and applepie count as one group of 2.
Show output
2 applesauce
1 banana
Show every copy of a duplicate group with a separator, long-form flag
sort wordlist.txt | uniq --all-repeated=separate--all-repeated is the long-form, more configurable version of -D: separate puts a blank line between groups; prepend (the other common mode) also puts one before the very first group.
Show output
apple
apple
apple
banana
banana
Same thing, with a separator before the first group too
sort wordlist.txt | uniq --all-repeated=prependprepend mode adds a leading blank line before every group including the first, which separate mode doesn't - a small formatting difference worth knowing before scripting against the output.
Show output
apple
apple
apple
banana
banana
Skip fields, long-form flag
printf '1 apple\n2 apple\n' | uniq --skip-fields=1--skip-fields is the long-form spelling of -f - same behaviour, more explicit in a script someone else will read later.
Show output
1 apple
Skip characters, long-form flag
printf 'xxapple\nyyapple\n' | uniq --skip-chars=2--skip-chars is the long-form spelling of -s.
Show output
xxapple