Score:0

Server

how to add multiples ips cidr with ipset save and restore options

acgbox

11/16/23, 1:30 PM

i have this loop to block IPs and CIDR with ipset/iptables:

# this is just an example. the actual list IPs/CIDR is very large
cat blockip.txt
13.31.0.254
cat blockcidr.txt
13.32.0.0/15

Loop:


#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset -A blacklist $ip
done
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

Note: I have always used the ipset -A option, but I don't know exactly what this option means, since it does not appear in "Man Ipset", and at this point, I'm assuming that add is the same as -A, since the output in both cases is the same.

#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset add blacklist $ip -q
done
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

out both cases:

sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xbc0136c8
Size in memory: 552
References: 0
Number of entries: 2
Members:
13.31.0.254
13.32.0.0/15

"It works fine", but I have read that adding the IPs and CIDR with ipset -A is very slow. Which is faster using ipset save and restore. But I don't understand how works, And my attempt is failed:

Note: I have not found an explanation of why it is faster to use the save/restore options, instead of add or -A

#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset add blacklist $ip # ??
    ipset save blacklist -f newblacklist.txt # ???
done
ipset restore -! < newblacklist.txt # ??
iptables -A FORWARD -m set --match-set blacklist dst -j DROP # ??

out:

sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xcb0e583b
Size in memory: 552
References: 0
Number of entries: 2
Members:
13.32.0.0/15
13.31.0.254

cat newblacklist.txt # out wrong
create blacklist hash:net family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xcb0e583b
add blacklist 13.32.0.0/15
add blacklist 13.31.0.254

I would appreciate any help (with a complete answer, including the proposed loop or corrections to my loop)

759

1 + 2

iptables

ipset

Martin

11/16/23, 1:52 PM

have you looked at the output of `ipset save`? I would create the empty sets with the proper options (f.e. `ipset create test hash:net family inet hashsize 1024`), use `ipset save > file`, and add the proper `add blacklist` lines to the saved file afterwards...

acgbox

11/16/23, 1:56 PM

I would appreciate it if you post your full answer with the sequence or corrections. Thank you

Score:1

Server

Martin

11/16/23, 2:21 PM

This worked for me:

# remove any old reference to the ipset
iptables -D FORWARD -m set --match-set blacklist dst -j DROP
ipset destroy

ipset create blacklist hash:net family inet hashsize 1024
ipset save > /tmp/ipset.txt
ipset destroy

cat blockip.txt blockcidr.txt | while read line; do
    if [ "${line:0:1}" = "#" ]; then
        continue
    fi
    echo "add blacklist $line" >> /tmp/ipset.txt
done

ipset restore < /tmp/ipset.txt
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

Please adapt the options in the create statement according to your needs - depending on the size of the set, these options are quite important. This further assumes that the file blockcidr.txt contains only lines like this:

#comment
1.2.3.4/20

afterwards, a restore works just fine.

update
the bottleneck of such a loop is always the creation of a subprocess, ipset in this case. If you have 10000+ entries, the executable ipset is being loaded into memory, options parsed for each line you would like to add...

My loop contains only bash-internal commands, so no executable must be loaded / executed - there is just being some text written into a file. And of course - a single call to ipset is much faster than 10000+ calls to ipset...

+ 14

acgbox

11/16/23, 2:30 PM

your proposal sounds reasonable, but it is incomplete, since you do not include the iptables command, it does not include the two lists (one with IPs and one with CIDR) and it is missing `maxelem 1000000`. And I would also like to know, why your loop is faster or better than mine. Thank you

Martin

11/16/23, 2:33 PM

I added some information why this loop is faster. you already have the appropiate iptables command, why should I copy & paste it?

acgbox

11/16/23, 3:07 PM

Very good clarification, but you have not fix your loop. I asked that "I don't understand how save/restore works", your loop doesn't answer this question. Maybe that's why I don't understand. PD: I don't know why you use comments (it doesn't exist in the original lists "#"), and why do you use a single list, when there are two lists

Martin

11/16/23, 3:08 PM

I used this loop to block country specific CIDRs - and my input files did have comments in their header, that is why I included it... If you do not need this - delete the complete `if` statement. The loop for the second input file would just be a copy & paste from the first one, using `hash:ip` as type instead of `hash:net`.

acgbox

11/16/23, 3:14 PM

I am now understanding your loop and I thank you for the explanation. But why can't I just use both files with hash:net (or merge them together) so I don't add more unnecessary lines to your loop? (in my loop I can do it). If your answer is that it is not possible, I appreciate you posting the complete sequence including the two files

Martin

11/16/23, 3:24 PM

ah, you want both input file inside one set... I misunderstood your intentions, sorry about that. of course you can read two files at once, I edited my answer accordingly!

acgbox

11/16/23, 3:27 PM

one other thing: `ipset destroy` is not working or is not in the right place. So, if there are new IPs or CIDRs in the lists, the "blacklist" will not be updated (error already exists). I was assuming you included this command to remove "blacklist" and recreate it every time you run the script ("ipset v7.15: Set cannot be created: set with the same name already exists" and "ipset v7.15: Error in line 3: Element cannot be added to the set: it's already added" so i need to include `-q` but this just hides the problem

acgbox

11/16/23, 3:31 PM

Note that even though it's not mentioned in the question, I can add more lists to the loop, like `wget http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz`

Martin

11/16/23, 3:40 PM

yes, the intention of the destroy had been to avoid the "already exists" error - it is probably failing because you cannot delete an ipset which is being referenced by an iptables rule. if that is the case, you could use `ipset flush` instead (which empties the set without deleting it). Yes, you could add more files to the loop.

acgbox

11/16/23, 4:16 PM

Great, but it should go to the beginning of the loop. And I also think you should include the options `-!` (to ignore errors) and `-q` (to hide the message that blacklist already exists, generated by `ipset create`), but I don't know where it will be more convenient to do these changes. Please rate this and put it in your script if necessary. (maybe `ipset create -q etc etc` and `ipset restore -! < /tmp/ipset.txt`)

Martin

11/16/23, 6:34 PM

I am no friend of silencing these errors / warnings; those exist for a reason... my proposition is to remove first the iptables rule referencing the ipset, read in the new ipset, and add the ipset again... this way, no errors should occur! I updated the answer accordingly.

acgbox

11/17/23, 1:30 AM

Last remark: ipset v7.15: Error in line 65538: Hash is full, cannot add more elements. I guess i have to increase `maxelem` (I don't know if i need to increase `hashsize` as well and I don't know what is the maximum allowed in both)

Martin

11/17/23, 8:06 AM

read [here](http://www.odi.ch/weblog/posting.php?posting=738) for useful parameter selection. I would rather recommend splitting your large dataset into multiple ipsets

acgbox

11/24/23, 7:29 PM

I think the script has an error. I made a new question: https://serverfault.com/questions/1116531/how-can-i-organize-the-ips-cidr-when-executing-restore-ipset

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: how to add multiples ips cidr with ipset save and restore options

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.