Compare new txt file with old txt file and remove all data that matches

I have a new file with the following data separated by a carriage return

a
a
b
c
d
d

I have an old file that is also separated by a carriage return

b
d

How do I remove b & d from the new file and remove one of the a's from the first file?

The desired output, separated by a carriage return, would be

a
c

I have tried sort -u which removes the b & d but also removes the a a. I have tried grep -vxFf however, there are duplicates from the new file.

Using awk, print only the lines of newfile that haven't previously occurred in either file:

awk '!(seen[$0]++ || NR==FNR)' oldfile newfile

If you can sort the files (which I assume as you said you tried sort -u), you can run comm oldfile.sorted newfile.sorted which will show the contents in three columns - old file only, new file only, both files. The -1, -2, -3 options allow you to suppress some of the columns, so comm -13 oldfile.sorted newfile.sorted | uniq should do what you want.

Read man grep and do something like:

grep -F -f oldfile -v newfile

In perl:

perl -ne 'print if ! ( $seen{$_}++ || $#ARGV eq 0 )' oldfile newfile

Or:

perl -ne '( $seen{$_}++ || $#ARGV eq 0 ) || print' oldfile newfile

Use comm:

comm -1 -2 <(sort -u old) <(sort -u new)

Though the grep answer works for you, sort and uniq can do the trick also.

Assuming you have the construct <(command) available (e.g. use bash). The construct will be replaced by a temporary file, containing the output of command.

This will work, with your example:

sort <(uniq new) old | uniq -u

Though more general would be:

sort <(uniq new) old old | uniq -u

Which also works if the old file contains lines that are not in new.

What happens is, that first the new file gets it duplicate lines removed. (This assumes that duplicates in the new file are adjacent. Otherwise replace <(uniq new) by <(sort -u new).)

Then the output of sort ... is taken as input to uniq -u.

uniq -u prints lines that are occurring only once.

Because the old file is presented twice to the sort ... command, none of the lines in old will make it to the output of uniq -u.

Also any lines common between old and new will not be present.

sort and uniq can be combined in several ways to perform mathematical set operations:

1. Set union: f U g

   sort f g | uniq

2. Set intersection: f ∩ g

   sort f g | uniq -d ## uniq -d only prints duplicate lines

3. Set difference: f \ g

   sort f g g | uniq -u ## uniq -u only prints unique lines

Note that the files f and g represent sets. So they must not have duplicate lines. If that is not the case, replace e.g. f by <(sort -u f).