Score:3

How to only remove duplicate lines if they're immediately after each other in the file

in flag
Cas

Lets say I have the following file:

$ cat test.txt
a
-----
b
-----
-----
c
-----
-----
-----
d
-----
e
-----
-----

Now I want to remove all the -----, but only if they're repeating after each other. So the result should look like this:

a
-----
b
-----
c
-----
d
-----
e
-----

I tried grep -Pvz -- "-----\n-----", but that didn't work.

Score:10
hr flag

That's exactly what the uniq command is made for:

NAME
       uniq - report or omit repeated lines

SYNOPSIS
       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION
       Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT
       (or standard output).

       With no options, matching lines are merged to the first occurrence.

So

$ uniq test.txt 
a
-----
b
-----
c
-----
d
-----
e
-----

Alternatively, you can use this sed one-liner 69. Delete duplicate, consecutive lines from a file (emulates "uniq") from Sed One-Liners Explained, Part III: Selective Deletion of Certain Lines and Special Applications

sed '$!N; /^\(.*\)\n\1$/!P; D' test.txt

which might be preferred if you want to edit test.txt in place (by adding the -i or --in-place option).

Score:0
ai flag

just do uniq filename.txt As the name states only unique lines are left and the repetitions are merged

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.