Sometimes people setup rate limiting using iptables.
The OpenSSH has the MaxStartups option that does some rate limiting on incoming clients. The default (at least on my computer) is 10:30:100.
man sshd_config
Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd(8) will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).
One other semi-common problem and cause of OpenSSH connection delays is related to an feature on the OpenSSH server that will attempt to do a reverse lookup of the incoming IP address on a connection attempt. I believe it needs this DNS functionality for compatibility with some of the older rhost compatible methods of authentication, functionality that I think almost nobody uses anymore. Anyway the DNS resolution feature will cause problems if the DNS resolvers is badly configured, are configured to use a broken resolver, or perhaps something about the reverse zone the client IP is connecting from is broken.
Ideally the answer is to fix DNS and make sure your DNS is always works without errors, and replies quickly. But if you don't require the DNS resolution, then a quick solution here is to stop the server from attempting to resolve names. Set UseDNS no in your sshd_config.
