Something is closing connections in my CentOS VMs - how to best troubleshoot?

I have a setup with 3 VMs (1 application server on CentOS6 and 2 database servers on CentOS7). The last 1-2 weeks we have had issues with timeouts when connecting to the database servers (and between the two servers that are in a cluster).

The database provider (Couchbase) can see from logs that the connections are forced closed:

WARN com.couchbase.endpoint - [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly

The logs also show that packages are dropped, like:

[warn] Interface ‘ens32’ (removedip) failures: RX:2863 / TX:0 - Details:
- RX packets:308,593,167 errors:0
dropped:2,863 overruns:0 frame:0

The VMs are hosted on the same host which is a VMware ESXi (version 6.5). So they should be able to have good connections to each other.

And what has changed over the last couple of weeks? Security updates on the VM OSes and the database server version (from 6.6.0 to 7.0.0). The database upgrade shouldn't change anything in the network but obviously is the reason why I first contacted the database provider...

Any ideas to find the culprit much appreciated :-)

Edit:

Following Camerons suggestion I just ran a short network trace and loaded it into Wireshark on my local machine. Then I opened the "Expert information" and got this: I need to say that there is an Nginx proxy server in front of the application server. It handles SSL and "lifts it off" before hitting the app. server. Just looking at the info I would expect the two "red" blocks to be related to requests coming from the outside - and not from the app. server to the database servers.

But I'm not really sure what to look for in the results? - and I guess I need to let it run a little longer - but perhaps without the information from the outside?

Edit 2

While sitting and looking at it the issue actually arose... - so I quickly started the tcpdump again. So the results may not contain the root cause - but should be more relevant than the first: The blocks I have expanded seem to be related to communication with one of the database servers.... :-)

But what do these results mean and how do I get closer to finding the cause?

welcome to Server Fault.

Given the age; CentOS 6 being out of support now, it would be very likely that you are suffering from SSL/TLS incompatibilies; assuming of course that you are connecting over that. We've certainly experienced plenty of such events over our time with RHEL6 as SSL2 etc. got progressively disabled by default. Similarly with various point-versions of Java (some point releases in 1.7 series were particularly fractious)

Another possible reason, seeing as you are running a CentOS workload on ESXi, is that you could be running shy of entropy, which causes blocking behaviour which can lead to timeouts and cluster issues, leading to connection aborts. Up to somewhere within Java 8, Java was particularly susceptible to this. You can judge if this is a problem for you by looking at /proc/sys/kernel/random/entropy_avail over time; if it gets to under 128 or so and doesn't bounce back, then you have entropy starvation. Common on a VM where there is no keyboard-mouse activity; you might try running an entropy gather daemon if this is the case.

BTW, I wouldn't conclude from those logs that something [else] is actively forcing those connections closed; its just that the connection closed at at time when one party wasn't expecting it to. This could be due to things like timeout, exceptions, process crash etc etc.

You say that the database server was upgraded... was that an OS upgrade from CentOS 6? Was the application also upgraded, or was it lifted and shifted?

Cheers, Cameron