High CPU usage with while loops running dd command

I'm using this command to read continuous data from named pipes:

{ while :; do dd iflag=fullblock iflag=nonblock bs=65536 count=1 2> /dev/null | redis-cli -x PUBLISH myChannel ; done } < myFifo

The problem is, the CPU usage goes very high even if I run 50 of those commands concurrently. That process is supposed to be long running, and a lot of them should work concurrently.

So, what is the reason and how to prevent it? Thanks.

If I'm not mistaken, you're using dd for buffering purposes ... A loop might not be needed in your case ... dd will continue reading and feeding the specified size blocks of data if you don't set a count(This instructs dd to exit after fulfilling the specified number of reads/writes) or iflag=nonblock(You’d want Blocking I/O to successfully initiate the read and keep reading from a named pipe with dd) and use it like so:

dd if=myFifo iflag=fullblock bs=65536 2> /dev/null | redis-cli -x PUBLISH myChannel

In this case, it should only exit when the end of the input file is reached(e.g. when the writer to the named pipe terminates/closes the pipe).

Or to keep the pipe constantly open waiting for writes, use it like so:

tail -c +1 -F myFifo | dd iflag=fullblock bs=65536 2> /dev/null | redis-cli -x PUBLISH myChannel

Or if your application expects end of stream/pipe(e.g. EOF or close_write ... Which is BTW not the best choice for a streaming application), use GNU parallel instead like so:

tail -c +1 -F myFifo | parallel --max-procs 1 -P 1 -j 1 --pipe --block 64k -k 'redis-cli -x PUBLISH myChannel'

This should resemble your loop but only where you need it to … It will do so in a rather controlled and resources aware way … It should also keep the named pipe constantly open even between writes, preserve every bit of the stream and shorten the pipeline.

The while : is spamming your CPU. Any command in a while :; loop will result in high CPU usage. For example:

while :; do echo foo > /dev/null; done

Or, even more blatant, a no-op:

while :; do true; done

The command you run is almost irrelevant: if the command itself doesn't take much time, then the while will cause a high CPU usage. The while : means that the loop will be relaunched as soon as the command ends, over and over again. And since this dd will finish almost instantaneously, that means it is being launched over and over, multiple times a second, threfore taking loads of CPU.

The solution is to add a small pause between invocations:

{ 
  while :; do 
    dd iflag=fullblock iflag=nonblock bs=65536 count=1 2> /dev/null | 
      redis-cli -x PUBLISH myChannel 
    sleep 0.1
  done 
} < myFifo

Adding even a 0.1 second sleep between runs will stop this from being a CPU hog.