Join Log in
Score:3
Philippe Hebert avatar Server

Google Cloud Engine - Network unreachable (with NAT)

jp flag
Philippe Hebert

Context & issue

We would like to do maintenance on one of our Compute Engine instance (update dependencies via apt update/upgrade), but said instance does not have access to the public internet, even if there is a NAT in the network:

sudo apt update

Err:1 http://security.ubuntu.com/ubuntu jammy-jellyfish-security InRelease                                             
  Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out
...

ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4096ms

We weren't able to figure out why the instance is unable to access the public internet; given the below information, can you help us figure out the cause?

Info

Instance details:

Zone: us-central1-c

Networking:

  • Public DNS PTR Record: None

Network interface:

name VPC network VPC subnetwork Primary internal IP address External IP address
nic0 default (project default VPC) default 10.128.0.1 None

Firewalls:

  • HTTP traffic: Off
  • HTTPS traffic: Off

Cloud NAT Gateway:

Name Region VPC network Router
project-nat northamerica-northeast1 default project-router

Cloud Router:

Name Region VPC network VPN Gateway
project-nat northamerica-northeast1 default None

Firewall rules:

name network direction priority src_ranges dest_ranges allow deny disabled
default-allow-icmp default INGRESS 65534 0.0.0.0/0 icmp FALSE
default-allow-internal default INGRESS 65534 10.128.0.0/9 tcp:0-65535,udp:0-65535,icmp FALSE
default-allow-rdp default INGRESS 65534 0.0.0.0/0 tcp:3389 FALSE
default-allow-ssh default INGRESS 65534 0.0.0.0/0 tcp:22 FALSE
server-allow-grpc-840d2afd default INGRESS 100 10.128.0.0/9 tcp:50051,tcp:50052 FALSE
server-deny-all-840d2afd default INGRESS 200 0.0.0.0/0 icmp,udp,tcp FALSE
server-ssh-iap-840d2afd default INGRESS 100 35.235.240.0/20 tcp:22 FALSE

Additional Firewall rules added to test if the firewall rules were the issue (without any success):

name network direction priority src_ranges dest_ranges allow deny disabled
ubuntu-repositories default EGRESS 10 (IPs of: security.ubuntu.com, archive.canonical.com, us-central1.gce.archive.ubuntu.com, etc.) tcp:80,tcp:443 FALSE
server-vm-egress-all-https default EGRESS 10 0.0.0.0/0 tcp:80,tcp:443,tcp:8443 FALSE

Instance network configuration:

cat /etc/hosts

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata

cat /etc/resolv.conf

nameserver 127.0.0.53
options edns0
search us-central1-c.c.project-name.internal c.project-name.internal google.internal

ifconfig

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.128.0.2  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4001:aff:fe80:2  prefixlen 64  scopeid 0x20<link>
        ether 42:01:0a:80:00:02  txqueuelen 1000  (Ethernet)
        RX packets 9167  bytes 4126195 (4.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10106  bytes 1160860 (1.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 640  bytes 61620 (61.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 640  bytes 61620 (61.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip rule list

0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default

ip route show table all

default via 10.128.0.1 dev ens4 proto dhcp src 10.128.0.2 metric 100 
10.128.0.1 dev ens4 proto dhcp scope link src 10.128.0.2 metric 100 
local 10.128.0.2 dev ens4 table local proto kernel scope host src 10.128.0.2 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev ens4 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::4001:aff:fe80:2 dev ens4 table local proto kernel metric 0 pref medium
ff00::/8 dev ens4 table local metric 256 pref medium

dig google.com

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34116
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       142.251.161.101
google.com.             300     IN      A       142.251.161.113
google.com.             300     IN      A       142.251.161.102
google.com.             300     IN      A       142.251.161.138
google.com.             300     IN      A       142.251.161.100
google.com.             300     IN      A       142.251.161.139

;; Query time: 19 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun May 07 21:28:29 UTC 2023
;; MSG SIZE  rcvd: 135

If you need any more information, let me know. Thanks a lot!

279
1 + 4
nat
John Hanley avatar
cn flag
John Hanley
What happens if you disable both EGRESS firewall rules? Unless you know exactly what you are doing, do not create EGRESS firewall rules. There are services running inside the VM that must talk to other Google Cloud services.
1
Reply
Kiran Kotturi avatar
nu flag
Kiran Kotturi
Can you check the below article written by Firdaus Ahmad , which will guide you in DNS resolver issue. https://dausruddin.com/could-not-connect-to-security-ubuntu-com80/
0
Reply
John Hanley avatar
cn flag
John Hanley
@KiranKotturi - This is not a DNS or DHCP problem as the system is resolving hostnames correctly. Notice: `security.ubuntu.com:80 (185.125.190.39)`.
0
Reply
Philippe Hebert avatar
jp flag
Philippe Hebert
@JohnHanley Kiran's solution down below solves the issue - once applied, both EGRESS rules can be removed without affecting the ability of the machine to connect to the internet. Thanks for your contribution :)
0
Reply
Score:2
Kiran Kotturi avatar Server
nu flag
Kiran Kotturi

As per the information provided, it is observed that the Compute Engine instance is in us-central1-c zone whereas Cloud NAT Gateway and Cloud Router are in northamerica-northeast1 region. It clearly shows that they are in different regions/zones. Also, please note that NAT gateway is regional and cannot connect to the VM instances which are in different regions.

As per the official documentation

Cloud NAT gateways are associated with subnet IP address ranges in a single region and a single VPC network. A Cloud NAT gateway created in one VPC network cannot provide NAT to VMs in other VPC networks connected by using VPC Network Peering, even if the VMs in peered networks are in the same region as the gateway.

If the issue persists even after fixing the NAT config, I recommend you to perform a connectivity test and validate whether the connectivity is broken.

Hope the above information is useful to you.

+ 1
Philippe Hebert avatar
jp flag
Philippe Hebert
Superb, that solves it! Thanks a lot Kiran! I'm a bit baffled that this deployment (which was part of an official Google template) was not working. Maybe back when it was created, this limitation of NAT gateways as region-bound was not a thing then.
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.