kubernetes pod running on google compute engine not able to access metadata service

Amit

1/3/23, 4:49 AM

I am trying to run google cloud python sdk from inside a k8 pod, running on google compute engine. There is a service account attached to the VM, which is giving it access to the secrets manager. I am able to access secrets manager from the host, however running the python sdk from k8 pod complains of not able to access the metadata service

>>> secret_id = 'unskript_test'
>>> name = client.secret_path(project_id, secret_id)
>>> response = client.get_secret(request={"name": name})
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/conda/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "Getting metadata from plugin failed with error: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Compute Engine Metadata server unavailable"
    debug_error_string = "{"created":"@1630634901.103779641","description":"Getting metadata from plugin failed with error: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Compute Engine Metadata server unavailable","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":90,"grpc_status":14}"
>

metadata.google.internal doesnt get resolved from the k8 pod

jovyan@jovyan-25ca6c8c-157d-49e5-9366-f9d57fcb7a9f:~$ wget http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
--2021-09-03 02:11:19--  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
Resolving metadata.google.internal (metadata.google.internal)... failed: Name or service not known.
wget: unable to resolve host address ‘metadata.google.internal’

However, host is able to resolve it

ubuntu@gcp-test-proxy:~$ wget http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
--2021-09-03 02:11:27--  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
Resolving metadata.google.internal (metadata.google.internal)... 169.254.169.254
Connecting to metadata.google.internal (metadata.google.internal)|169.254.169.254|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-09-03 02:11:27 ERROR 403: Forbidden.

How can i make the pod resolve metadata.google.internal?

318

0 + 0

google-compute-engine

kubernetes

Bakul Mitra

7/23/23, 4:55 AM

Hello @Amit. Was your problem resolved? If yes, can you mention the steps you have taken to solve the problem. Or if the given answer helped can you accept or upvote it..

Score:0

Server

Amit

10/5/23, 7:31 PM

The issue was with microk8s not copying host etc hosts entry to the pods. Once we moved to k3, it got resolved

+ 0

Score:0

Server

Maciek

1/22/23, 12:37 PM

This is caused by Kubernetes pod not being able to resolve the metadata.google.internal DNS name to proper IP. You host machine probably has an entry in /etc/hosts that hardcodes that domain to the IP: 169.254.169.254.

You should be able to replicate that in your Pod by modifying its /etc/hosts file.

Just remember that this will only work on VMs running on GCP. Outside, the IP address 169.254.169.254 is just another IP address that has not special meaning.

EDIT: Just checked /etc/hosts on one of my GCP VMs, and here's what I found:

$ cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata

So just try copying that last line to your pods /etc/hosts.

0 + 0

Amit

7/24/23, 2:22 PM

Thanks @maciek. The issue was with microk8s not copying host etc hosts entry to the pods. Once we moved to k3, it got resolved

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: kubernetes pod running on google compute engine not able to access metadata service

TH: พ็อด kubernetes ที่ทำงานบนเครื่องมือคำนวณของ Google ไม่สามารถเข้าถึงบริการข้อมูลเมตา

RO: Pod kubernetes care rulează pe motorul de calcul Google nu poate accesa serviciul de metadate

RU: Модуль kubernetes, работающий на вычислительном движке Google, не может получить доступ к службе метаданных

VI: nhóm kubernetes chạy trên công cụ tính toán của google không thể truy cập dịch vụ siêu dữ liệu

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.