Score:0

Server

How can I organize the IPs/CIDR when executing restore ipset?

acgbox

11/24/23, 4:25 PM

I have this bash script of ipset v7.15 (run in ubuntu 22.04), courtesy of Martin (that I have made some modifications for this question):

ipset create -! blacklist hash:net family inet hashsize 1024
ipset save -! > /tmp/ipset.txt

cat list.txt | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n -k 5,5n -k 6,6n -k 7,7n -k 8,8n -k 9,9n | while read line; do
    echo "add blacklist $line" >> /tmp/ipset.txt
done
ipset restore -! < /tmp/ipset.txt

content of list.txt:

125.74.0.0/15
1.0.132.249
125.73.0.0/16
130.255.128.0/20
125.76.0.0/17

out /tmp/ipset.txt:

create blacklist hash:net family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6
add blacklist 1.0.132.249
add blacklist 125.73.0.0/16
add blacklist 125.74.0.0/15
add blacklist 125.76.0.0/17
add blacklist 130.255.128.0/20

out ipset -L (the problem):

sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6
Size in memory: 696
References: 0
Number of entries: 5
Members:
125.74.0.0/15
1.0.132.249
125.73.0.0/16
130.255.128.0/20
125.76.0.0/17

As you can see, the IPs/CIDRs in the blacklist output are messed up (no sort).

Ipset has this option:

-s, -sorted
Sorted output. When listing or saving sets, the entries are listed sorted.

But I tried it with the following command and the output is the same (without sort. Maybe I'm not using it correctly):

ipset restore -! -s < /tmp/ipset.txt

And if I remove -! option from the restore command, I get the following error:

ipset v7.15: Error in line 1: Set cannot be created: set with the same name already exists

So, i think my script has an error, since it adds a line that I think shouldn't be there:

create blacklist hash:net family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6

then I can remove the save line to avoid the error, leaving it like this:

ipset -! create blacklist hash:net family inet hashsize 1024
cat list.txt | while read line; do
    echo "add blacklist $line" >> /tmp/ipset.txt
done
ipset -s restore -f /tmp/ipset.txt

But the output remains the same, no-sort, when I run ipset -L, so the -sorted (-s) option is not working in restore

How can I fix this so that the output is organized?

167

1 + 4

linux

ipset

Martin

11/25/23, 9:19 AM

your set is a hash! Read [here](https://en.wikipedia.org/wiki/Hash_function) about what a hash is and what that means... of course, the set is organized internally by hash value, so that a check (is IP xy inside this set) always runs fast... and my manpage concerning `-s` reads: `Sorted output. When listing sets entries are listed sorted. Not supported yet.` If you really care about a sorted output, do a `ipset list | sort`!

acgbox

11/25/23, 1:19 PM

I haven't found "man ipset" that says the `-sorted` option "Not supported yet" (I'm working with v7.15). please provide the link where it says that

Martin

11/25/23, 1:25 PM

[here](https://linux.die.net/man/8/ipset) - the outcome of the sorted option (if it is supported or not) might depend on the kernel version you are using.

acgbox

11/25/23, 1:43 PM

mmm. You are right "not supported". I'm using kernel 5.15.0-53-generic (from Ubuntu 22.04) and ipset v7.15 and `-sorted` doesn't work, so I'm assuming it's for newer versions. Additionally, doing `sort` slows down these rules and is not recommended. I was just curious. Thanks. PS: If you want you can expand your answer to select it as correct.

Score:1

Server

Martin

11/26/23, 7:48 AM

Some background information on ipset: This is a command line tool to manipulate datasets which resize within the kernel memory. As such, ipset is just a wrapper to call the actual code, which resides inside the kernel - Including the code to list the contents of a set. Here comes the tricky part:

You passed an option that you would like to have the output of the set in a sorted manner - but the kernel module responsible for this did not know that option an simply ignored it. All functions which are executed in kernel context must be fast and could freeze your system if they take too long. That is why the kernel developer are quite picky when code is not strictly necessary (sorting is expensive in terms of CPU time and not necessary - this could be done in userspace, too).

I do not know what kernel version is required to "enable" the sorted output of a set, I tested it with kernel version 6.0.9, where the option to sort the output did what it should. It is a shame that they removed the hint on some man pages, here is a version of the man page which still has that hint.

The set being in such an order is not an error, it is actually a feature: in a list, you would have to go through every item to check if the incoming IP is part of that list. This makes every lookup dependent of the size of your list. A hash is different: A function is used to compute a hash value out of your data (the IP in this case). This hash value is used as reference on the position inside your data - if there is an entry there, the IP is inside that list.

This way of organizing your set makes the time required to lookup a value constant - independent of the size of your set! A fast response time is always important when it comes to network traffic.

I hope this helps to understand things better...

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How can I organize the IPs/CIDR when executing restore ipset?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.