Score:0

How to "scp" files from a source server and equally divide them among three destination servers?

ki flag

I am looking for a script to copy files from a source Server directory /opt/Test to be equally divided between three destination Server directories with non sequential order.

For instance I want to copy some files from source Server-A/ directory to destination Server-B, destination Server-C directory and copy the remaining files to destination Server-D directory.

I want something like this :

Source Server

**Server-A/**
├── file0
├── file1
├── file2
├── file3
├── file4
├── file5
├── file6
├── file7
├── file8
└── file9

Destination Servers

**SERVER-B**
├── file0
├── file3
├── file6
└── file9

**SERVER-C**
├── file1
├── file4
└── file7

**SERVER-D**
├── file2
├── file5
└── file8

mondotofu avatar
cn flag
rsync is a good tool for synchronizing files between two nodes, but you'll need some logic around your choices for copying the files. For example, when you say "equally divided" does this mean by file size, date, name, or some other measure? Is this a one-time hack or do you intend to schedule this for repeated operation? Are duplicates allowed?
sunman avatar
ki flag
@FedKad equally divide mean , sending files among all server in a such a way that one to Server-1 second file to Server-2 and third file to Server-3 then fourth file to Server-1 and so on..No duplication allowed
Raffa avatar
jp flag
I added an answer below that I hope will help … but consider mounting remote directories as local shares on server-A and copy to them locally with e.g. just `cp` if this is an option as this would be more efficient and much faster.
Score:1
jp flag

Using scp to copy a file from the local-host to a remote-host goes like this:

scp file username@remote-host:/remote/directory/

Where file is the local file you want to copy to the remote host, username is a valid username on the remote host, remote-host is the actual address/IP of the remote host and /remote/directory/ is the full path of the destination directory on the remote host ... You'll need to enter a password every time you copy a file to a remote host unless you set a password-less "SSH Key Authentication".

To achieve what you want you will need to prepare and test three scp commands(One for each remote server) like so:

scp file username@Host-B:/remote/directory/

scp file username@Host-C:/remote/directory/

scp file username@Host-D:/remote/directory/

These three command lines are examples using scp which should work fine for your use case but you can use any other tool like e.g. rsync.

Note that scp and other remote file copying utilities are prone to failure due to bad/dropping connection to the remote server ... So I would advise also to check the exit status of scp(It exits 0 on success and >0 otherwise) with e.g. until [ "$s" = 0 ]; do ... done where $s will hold the exit status of the lastly executed scp command so that if that command did not succeed in copying the file to one of the remote servers then the script will attempt to copy that file to the next server ensuring all files are copied to working servers even if one or more of them has connection issues.

Then use them in a for loop while alternating between them like so:

#!/bin/bash

# This script should be run on Server-A

i=1
for f in /opt/Test/*; do
  s=1
  until [ "$s" = 0 ]; do
      if [ "$i" = 1 ]; then
        scp "$f" username@Host-B:/remote/directory/
        s="$?"
        i=2
      elif [ "$i" = 2 ]; then
        scp "$f" username@Host-C:/remote/directory/
        s="$?"
        i=3
      elif [ "$i" = 3 ]; then
        scp "$f" username@Host-D:/remote/directory/
        s="$?"
        i=1
      fi
    done
  done

Alternatively, consider mounting remote directories from server-B, server-C and server-D as local shares on server-A and copy to them locally with e.g. just cp if this is an option as this would be more efficient and much faster ... Assuming you mount those on mount-points e.g. /mnt/B/, /mnt/C/ and /mnt/D/ then the script would need to be modified to use cp like so:

#!/bin/bash

# This script should be run on Server-A

i=1
for f in /opt/Test/*; do
  s=1
  until [ "$s" = 0 ]; do
      if [ "$i" = 1 ]; then
        cp -- "$f" /mnt/B/
        s="$?"
        i=2
      elif [ "$i" = 2 ]; then
        cp -- "$f" /mnt/C/
        s="$?"
        i=3
      elif [ "$i" = 3 ]; then
        cp -- "$f" /mnt/D/
        s="$?"
        i=1
      fi
    done
  done
sunman avatar
ki flag
following command not working elif [ "$i" = 2 ]; then scp "$f" username@Host-C:/remote/directory/
sunman avatar
ki flag
This script not working Properly; ""i=2 elif [ "$i" = 2 ]; then scp "$f" username@Host-C:/remote/directory/" This one is not working as some files are not copied for this condition and thus not copied fully files For example if i generated files from 1 to 10 .. file 2 and 8 are not copied for this condition
Raffa avatar
jp flag
@sunman This is most likely because of bad connection issues to server-c … When I have time I will add checking the exit status of `scp` and ensure the files are all copied to the script and update the answer … Meanwhile consider the alternative solution in my earlier comment under your question :-) https://askubuntu.com/questions/1444938/looking-for-bash-script-for-copying-files-from-a-source-server-directory-and-equ#comment2521725_1444938
sunman avatar
ki flag
thank you so much its working:-)
sunman avatar
ki flag
Is it possible if it check any destination server not available then it send to other available server ?
Raffa avatar
jp flag
@sunman Yes, possible ... Please check the updated answer.
sunman avatar
ki flag
i have performed this on folder.. From source Folder A, to destination Folder-B Folder-C and Folder-D it worked for Folder-B and Folder C and duplicate copy to Folder-D
Raffa avatar
jp flag
@sunman The only possible way I can think of that will result in the same file being copied to more than one destination directories is if you run the script more than one time on the same source and destination directories … If you start with empty destination directories and run the script only once, all the files from the source directory will be copied with no duplicates in the destination directories … of course those destination servers/directories with connection problems will contain fewer files as files will be copied to the other working servers.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.