how can I get unique ip addresses from /var/log/auth.log?

How do I only show the unique IP addresses in /var/log/auth.log ? I have a server getting hit constantly, and while I have set up fail2ban, the requests keep coming. So I started writing to ISPs to get the accounts shutdown, which might help a little, but there are so many attacks.... Anyway,

I used this command to sort the hits from today: $grep -e 'ruser \| rhost' auth.log | grep -e 'Jun 9' | sort -k 14 | less its thousands of attacks... or brute force attempts.

Heres some relivent lines from auth.log:

...
Jun  9 10:08:35 nan sshd[23397]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53
Jun  9 10:08:51 nan sshd[23401]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53
Jun  9 10:09:07 nan sshd[23428]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53
Jun  9 10:01:59 nan sshd[23216]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:02:52 nan sshd[23238]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:03:08 nan sshd[23246]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:03:40 nan sshd[23256]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:04:14 nan sshd[23270]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:04:31 nan sshd[23276]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:05:23 nan sshd[23298]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:05:55 nan sshd[23315]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:06:49 nan sshd[23343]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:07:47 nan sshd[23375]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:08:02 nan sshd[23383]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:09:47 nan sshd[23440]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:10:02 nan sshd[23444]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=root
Jun  9 10:09:27 nan sshd[23436]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.109.161.53  user=sshd
Jun  9 05:27:00 nan sshd[20293]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.134.44.86
....

I would like to make a list of the unique IP addresses in auth.log.

I tried using gawk -F: '{ print $14 }' auth.log | uniq -c

but I guess I'm doing something wrong. I just want a sorted list of IP addresses from today. how do I do that?

uniq only detects adjacent duplicates - you'd need to sort first

awk '{print $14}' auth.log | sort | uniq

or use the sort command's own -u option

awk '{print $14}' auth.log | sort -u

or (since you're already using awk)

awk '!seen[$14]++ {print $14}' auth.log

(The rhost= entry is the 14th whitespace delimited field - no need for -F:)

With grep

I see you're struggling a bit with grep matching patterns as you appear to need to match some patterns before printing unique IPs ... So, ...:

I would use grep's option -o to only print the matched part and enable Perl style regular expressions with -P and use:

• Jun to match Jun and then ...
• \s+ to match one or more spaces and then ...
• 9 to match 9 and then ...
• .* to match anything until ...
• ruser= is matched and then ...
• .* to match anything until ...
• rhost= is matched and then ...
• exclude all the above from printing with Perl's \K and then ...
• match an IP with \d+\.\d+\.\d+\.\d+ where \d+ matches one or more digits and \. matches a literal dot and then ...
• pipe the output to sort -u to print only unique IPs like so:

grep -oP 'Jun\s+9.*ruser=.*rhost=\K\d+\.\d+\.\d+\.\d+' auth.log | sort -u

With gawk

Or since you appear to have GNU AWK installed, you can do it all(including sorting ... utilizing Predefined Array Scanning Orders) with gawk like so:

gawk 'BEGIN {
    FS = "[= ]*"
    PROCINFO["sorted_in"] = "@ind_num_asc"
}

/Jun\s+9/ && /ruser/ && /rhost/ {
    a[$18]++
}

END {
    for (i in a) {
        printf "%s\n", i
    }
}' auth.log

$ egrep -o '[1-2]?[0-9][0-9]?\.[1-2]?[0-9][0-9]?\.[1-2]?[0-9][0-9]?\.[1-2]?[0-9][0-9]?' /var/log/auth.log \
|sort -u

I made up the regex pattern, but any regex pattern that matches IP address (there are lots online) should be allowed.

Useful for searching any logs for IP addresses.

Raffa's answer is a bit cleaner, though the first digit in each segment of an IP can only a 1 or 2. (ignoring IPv6 addresses).