How do I pipe logs from multiple linux instances over the network into one file?

Chip

4/25/24, 11:32 PM

I have a shell script running on hundreds of linux machines that outputs a csv summary line at the end. I need to collect that into one file to review the summary of results. Is there a way to transfer that csv line over the network to one single machine that pipes them all to a single file?

Maybe something like a simple http server that append >> prints all input to a file and then the script can run a curl command to post it to that server?

1 + 2

linux

doneal24

4/26/24, 12:55 AM

Do you actually have an httpd daemon set up on each client machine? If not, you might consider something like [parallel ssh](http://code.google.com/p/parallel-ssh/) although this could result in interleaving lines in the csv. Alternative might be `for host in $clients; do ssh $host cat /foo/bar.csv; done > aggregate.csv`. Possibly time consuming if you have thousands of hosts. Also assumes that you have pki authentication among hosts. Other alternative is if the hundreds of systems have a shared file system, put all of the csv's in a single directory and just `cat` into a single file.

Chip

4/27/24, 1:19 AM

It's many posting to one rather than one to many. I need the hundreds of instances running the script to be able to call back their output line at the end to the one instance that can then dump them to a single file together. Order is not important.

Score:1

Server

doneal24

4/27/24, 2:55 PM

A simple solution to have hundreds of systems ship an application's output back to a single would be to use netcat.

One the central server, put netcat into a infinite loop to receive input.

while true; do nc -l 4444; done | tee foo.csv

Then on each of the application hosts run

app.exe | nc foo.example.com 4444

Maybe not very robust but it would get the job done.

For a more reliable solution, set up rsyslog on the central server. Then on each of the app hosts use logger --server foo.example.com "your message here". The messages posted through logger will be recorded in a file by rsyslog for further processing. Possibly more difficult to set up but easier in the long run.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How do I pipe logs from multiple linux instances over the network into one file?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.