Score:0

munin not receiving full plugin output over network but plugins work locally (munin-run)

mu flag

I have the strangest problem with munin:

When I use telnet or netcat to directly connect from my munin master to the target node or when munin-cron does its thing, some plugins only send the first line of output or sometimes no output at all. munin-cron is then waiting for the full reply to finish and eventually times out.

Here is an example: I connect from the master to the target node and fetch the load plugin. However when I try fetch the memory plugin the first line of output appears and the cursor just blinks forever:

enter image description here

However running the plugin (fetch and config) locally with munin-run works as it should:

enter image description here

enter image description here

And when looking into the memory plugin, after the first line of output it simply sends more lines... there is nothing being executed in between the lines in the config part. So i see no reason why it could possibly hang at this point?!

my %mems;
&fetch_meminfo;

if ($ARGV[0] and $ARGV[0] eq "config") {
    print "graph_args --base 1024 -l 0 --upper-limit ",$mems{'MemTotal'},"\n";
    print "graph_vlabel Bytes\n";
    print "graph_title Memory usage\n";

What could be the reason for this behaviour? It kind of started suddenly, I did not change anything on the two servers and I rebooted them both already but the problem persists. So far its the "diskstats" and "memory" plugins. Other plugins like load, df. etc. are working fine...

How could I debug this? I already have the log_level up to 4 on all nodes but the target node just tells me that the master is connecting to it and thats it. The master is only logging the incomplete output and eventually the timeout.

Oh and I have multiple servers all of the same configuration and distro and they're working just fine. Just this one server somehow does not "fully answer" munin.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.