Score:0

Why does the login node connect to external networks but allocated compute node fail in Slurm-GCP?

nl flag

I've noticed that connecting to the internet from the allocated compute node via Slurm-GCP keeps failing. For example, using wget from the login node works successfully:

[me@gcp-login0 ~]$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz
--2023-05-11 19:06:34--  https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 111.111.1.111, 111.111.11.111, 111.111.111.111, ...
Connecting to cdn.kernel.org (cdn.kernel.org)|111.111.1.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 102167060 (97M) [application/x-xz]
Saving to: ‘linux-4.17.2.tar.xz’

100%[======================================>] 102,167,060  277MB/s   in 0.4s   

2023-05-11 19:06:35 (277 MB/s) - ‘linux-4.17.2.tar.xz’ saved [102167060/102167060]

But on a single allocated GPU, wget stalls and fails:

[me@gcp-compute-0-0 ~]$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 111.111.1.111, 111.111.11.111, 111.111.111.111, ...
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.1.176|:443... 
failed: Connection timed out.

Accordingly, I was wondering if there's a way to solve this network issue for compute nodes on Slurm-GCP? I tried modifying firewall settings and VPC network details, but it seems to only affect the login node and unable to target the compute node settings on GCP.

Score:0
st flag
As a workaround by Goli Nikitha, trying to SSH into login and use srun --pty $SHELL which takes you to SSH page of controller. I followed this documentation and this works.   Attaching here as well a reference on Diagnosing and resolving common issues in Fluid-Slurm-GCP
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.