We're using Consul v1.11.1 KV to store some read-mostly data(about 50 bytes per key) . I'm doing performance test on Consul get recently. Frist I use a single server(8CPU Cores,2.4GHz; 16GB RAM; 300G SSD)
wrk -t8 -d30 -c200 http:8500/v1/kv/aa Running 30s test @ http://10.190.4.251:8500/v1/kv/aa
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.86ms 2.99ms 55.02ms 88.33%
Req/Sec 10.71k 2.00k 19.79k 69.19%
2565410 requests in 30.09s, 0.93GB read
Requests/sec: 85251.41
Transfer/sec: 31.79MB max cpu 700%
Then I create a cluster with other same configuration server(8CPU Cores,2.4GHz; 16GB RAM; 300G SSD) and build a haproxy for load balancing enter image description here
at this time max cpu 300% per server. the result
Running 2m test @ http://10.190.5.152:8999/v1/kv/aa 8 threads and 200 connections Thread Stats Avg Stdev Max +/- Stdev Latency 5.38ms 1.72ms 212.79ms 94.57% Req/Sec 4.68k 388.44 7.27k 84.26% 4474792 requests in 2.00m, 1.63GB read Requests/sec: 37260.15 Transfer/sec: 13.89MB
i increase connections but find Non-2xx response.
Running 2m test @ http://10.190.5.152:8999/v1/kv/aa 8 threads and 250 connections Thread Stats Avg Stdev Max +/- Stdev Latency 6.64ms 1.95ms 213.80ms 93.03% Req/Sec 4.70k 466.78 10.65k 82.20% 4485786 requests in 2.00m, 1.63GB read Socket errors: connect 0, read 129, write 0, timeout 0 Non-2xx or 3xx responses: 64 Requests/sec: 37353.78
Higher performance of a single server than a cluster. I don't kown why? Am I using it wrong?