I have accepted solution of @sebres but I would like to add some gotchas.
For iptables-allports banaction, the reject blocktype can have spaces inside. You need to quote that.
Example:
[sshd]
banaction=iptables_allports[blocktype="REJECT --reject-with icmp-port-unreachable"]
Second interesting thing: both the banaction and the jail config have a parameter called "protocol". I was first confused when the configuration below was not throwing any errors, but it did not block UDP requests:
[named-ddos]
banaction=iptables_allports[blocktype=DROP,protocol=all]
It happened because I was missing the protocol=all setting from the jail. You need to specify protocol=all at the jail level:
[named-ddos]
banaction=iptables_allports[blocktype=DROP,protocol=all]
protocol=all
The reason for this is that the named-ddos section creates a new chain in iptables, and the banned ips are creating rules inside that chain. If you don't specify protocol=all at the jail level, then the chain will be defined like this:
Chain INPUT (policy DROP 22 packets, 952 bytes)
pkts bytes target prot opt in out source destination
1371 229K named-ddos tcp -- * * 0.0.0.0/0 0.0.0.0/0
It is true, that the banaction will create rules with proto=all inside the chain, but the chain itself won't be used for non-tcp packets. The conclusion is that you need to specify protocol=all in both the jail level and in the banaction (if it supports it), otherwise it won't work.