I have the strangest problem with munin:
When I use telnet or netcat to directly connect from my munin master to the target node or when munin-cron does its thing, some plugins only send the first line of output or sometimes no output at all. munin-cron is then waiting for the full reply to finish and eventually times out.
Here is an example: I connect from the master to the target node and fetch the load plugin. However when I try fetch the memory plugin the first line of output appears and the cursor just blinks forever:
However running the plugin (fetch and config) locally with munin-run works as it should:
And when looking into the memory plugin, after the first line of output it simply sends more lines... there is nothing being executed in between the lines in the config part. So i see no reason why it could possibly hang at this point?!
my %mems;
&fetch_meminfo;
if ($ARGV[0] and $ARGV[0] eq "config") {
print "graph_args --base 1024 -l 0 --upper-limit ",$mems{'MemTotal'},"\n";
print "graph_vlabel Bytes\n";
print "graph_title Memory usage\n";
What could be the reason for this behaviour? It kind of started suddenly, I did not change anything on the two servers and I rebooted them both already but the problem persists. So far its the "diskstats" and "memory" plugins. Other plugins like load, df. etc. are working fine...
How could I debug this? I already have the log_level up to 4 on all nodes but the target node just tells me that the master is connecting to it and thats it. The master is only logging the incomplete output and eventually the timeout.
Oh and I have multiple servers all of the same configuration and distro and they're working just fine. Just this one server somehow does not "fully answer" munin.