I'm trying to get nested KVM working in Google Cloud, but I'm having trouble with Centos 7 dropping traffic that's returning through IP Tables.
Centos 7 forms a virtual router (VR), which sits at the front of the group of devices that sit in KVM. The OS running on the cloud is Ubuntu 18 (I've also tried 16 and 20 - same results). My IP address on br0 (on Ubuntu) is 172.30.7.1 and the IP address on the outbound interface on the Centos VR is 172.30.7.100. A device behind the Centos VR is making a request to a server and getting a response back. The first few packets come through okay, but some packets are dropped. I can't figure out why, but this precipitates retransmits from the server, none of which make it to the device.
This is what I'm seeing on the VR coming back from the server:
06:05:32.671293 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [S.], seq 3432102555, ack 164728512, win 65535, options [mss 1430,sackOK,TS val 3630073863 ecr 5296823,nop,wscale 8], length 0
06:05:32.671308 IP 34.84.96.34.bc.googleusercontent.com.https > device.38298: Flags [S.], seq 3432102555, ack 164728512, win 65535, options [mss 1430,sackOK,TS val 3630073863 ecr 5296823,nop,wscale 8], length 0
06:05:32.672567 IP localhost.localdomain.38298 > 34.84.96.34.bc.googleusercontent.com.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 5296826 ecr 3630073863], length 0
06:05:32.673732 IP device.38298 > 34.84.96.34.bc.googleusercontent.com.https: Flags [P.], seq 1:235, ack 1, win 229, options [nop,nop,TS val 5296827 ecr 3630073863], length 234
06:05:32.673745 IP localhost.localdomain.38298 > 34.84.96.34.bc.googleusercontent.com.https: Flags [P.], seq 1:235, ack 1, win 229, options [nop,nop,TS val 5296827 ecr 3630073863], length 234
06:05:32.675099 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], ack 235, win 261, options [nop,nop,TS val 3630073867 ecr 5296827], length 0
06:05:32.675111 IP 34.84.96.34.bc.googleusercontent.com.https > device.38298: Flags [.], ack 235, win 261, options [nop,nop,TS val 3630073867 ecr 5296827], length 0
06:05:32.676296 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 1408
06:05:32.676307 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 10
06:05:32.676312 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1419:2827, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 1408
06:05:32.676317 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 2827:2837, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 10
06:05:32.676319 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 2837:4245, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 1408
06:05:32.676325 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 4245:4255, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 10
06:05:32.676330 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 4255:5663, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 1408
06:05:32.676332 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [P.], seq 5663:5672, ack 235, win 261, options [nop,nop,TS val 3630073868 ecr 5296827], length 9
06:05:32.684887 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 4255:5663, ack 235, win 261, options [nop,nop,TS val 3630073877 ecr 5296827], length 1408
06:05:32.684906 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [P.], seq 5663:5672, ack 235, win 261, options [nop,nop,TS val 3630073877 ecr 5296827], length 9
06:05:32.892016 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630074084 ecr 5296827], length 1408
06:05:32.892051 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630074084 ecr 5296827], length 10
06:05:33.299979 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630074492 ecr 5296827], length 1408
06:05:33.300007 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630074492 ecr 5296827], length 10
06:05:34.147973 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630075340 ecr 5296827], length 1408
06:05:34.147996 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630075340 ecr 5296827], length 10
06:05:35.811929 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630077004 ecr 5296827], length 1408
06:05:35.811963 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630077004 ecr 5296827], length 10
06:05:37.962059 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38272: Flags [.], seq 1:1409, ack 236, win 261, options [nop,nop,TS val 1904087777 ecr 5295821], length 1408
06:05:37.962098 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38272: Flags [.], seq 1409:1419, ack 236, win 261, options [nop,nop,TS val 1904087777 ecr 5295821], length 10
06:05:39.076057 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1:1409, ack 235, win 261, options [nop,nop,TS val 3630080268 ecr 5296827], length 1408
06:05:39.076091 IP 34.84.96.34.bc.googleusercontent.com.https > localhost.localdomain.38298: Flags [.], seq 1409:1419, ack 235, win 261, options [nop,nop,TS val 3630080268 ecr 5296827], length 10
I'm not sure what to do from here. I've tried changing the Ubuntu Instance that KVM is running on to version 16 and version 20, but it hasn't made any difference. I've also tried changing the CentOS 7 instance to an Ubuntu image, but it showed the same behavior.
Any suggestions are welcome.
