k3s kubectl logs times out after 50 seconds

Jorge Yanes Diez

2/14/24, 2:23 PM

TLDR: Depending on my k3s version, kubectl logs -f drops back to the console (timeouts?) exactly after 50 seconds without new log entries, instead of remaining attached until a Ctrl-C is issued. I can't seem to find where this is configured or why it happens.

OS: Ubuntu Server 22.04 (virtual machine) with 6GB RAM that can grow up to 8GB and plenty of disk space. VM is vanilla (no weird configuration applied).

EDIT 1: @mdaniel - Network topology is the simplest case possible: Single VM with nothing explicitly configured. Default DHCP over TCPIPv4 to connect to the Internet, and internally just using default networking (both TCPIPv4 and v6 enabled as it's the default in Ubuntu Server 22.04). The default k3s installation instructions set up a single node that acts both as a controller and an agent. Again, no explicit configuration statements used during the k3s install - using defaults for everything, thus the API server and kubectl are on the same machine running locally. I am SSHing to the machine itself to use kubectl, so no external networking problems are causing the issue.

EDIT 2: I am using the kubectl binary installed alongside k3s, not using the snap.

All versions of k3s up to and including v1.23.6+k3s1 worked perfectly for me: kubectl logs -f remains attached. From version v1.23.7+k3s1 onwards, the 50 second timeout drops me back to the console.

Note: I don't need to install anything on k3s to reproduce this - It happens even with the pods in the kube-system group on a completely new installation of a fresh VM.

This is a big issue for me, because I'm trying to run AWX which seemingly depends on this command to show me the result of my ansible playbook runs. As soon as I execute any playbook that takes longer than 50 seconds to log anything to the screen, my deployments fail without any useful information.

I have read all the release notes, man pages, Github issues and installation manuals I can - for all k3s, kubernetes and AWX, and I can't seem to find any references to this behavior. I have even searched inside the code repos in github and can't seem to find anything useful.

Does anybody have any information that can help me out? Until I can figure this out I'm stuck with k3s v1.23.6+k3s1 which is already quite old now.

Thank you for your time reading this entry.

307

0 + 3

kubernetes

k3s

kubectl

mdaniel

2/15/24, 3:33 AM

Without knowing the network topology between your kubectl and the api-server it's hard to generalize -- what investigation have you already done and what was its outcome? Don't comment, instead [edit your question](https://serverfault.com/posts/1122902/edit) and include those details

mdaniel

2/16/24, 4:59 AM

Then I only have two other hypotheses: maybe kubectl is running from a snap, since this is ubnutu; or, alternatively, have you already dialed up the verbosity to see what the traffic between kubectl and k3s has to say (`kubectl --v=10 logs -f $whatever 2>logs.err | tee logs.out`)

Veera Nagireddy

2/16/24, 1:58 PM

See [Github issue](https://github.com/k3s-io/k3s/issues/2829), which may help to resolve your issue.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: k3s kubectl logs times out after 50 seconds

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.