Overlooking your searching/copying methods as they appear irrelevant to the error you get.
Okay! ... I had never tried this before ... But, guess what ... I couldn't resist and have tried it a few minutes ago, and here are the results.
$ # Make mount points
$ mkdir fuse_mount1 fuse_mount2
$ # Create 2000 files of 1M each
$ truncate -s 1M file{0000..1999}
$ # Zip each 1000 files in a different archive
$ zip archive1 file{0000..0999}
$ zip archive2 file{1000..1999}
$ # Mount both the archives
$ fuse-zip archive1.zip fuse_mount1/
$ fuse-zip archive2.zip fuse_mount2/
$ # Copy "fuse_mount1" under "fuse_mount2"
$ cp -n -a -r ./fuse_mount1 ./fuse_mount2
$ # Unmount the two fuse filesystems
$ fusermount -u fuse_mount1/
$ fusermount -u fuse_mount2/
Which needed around 1G memory of my test machine's RAM of 20G ... So, nothing out of the normal.
So, I changed the formula:
$ # remove previous files and archives
$ rm file* *.zip*
$ # Create 200 files of 100M each
$ truncate -s 100M file{000..199}
$ # Zip each 100 file in a different archive
$ zip archive1 file{000..099}
$ zip archive2 file{100..199}
$ # Mount both the archives
$ fuse-zip archive1.zip fuse_mount1/
$ fuse-zip archive2.zip fuse_mount2/
$ # Copy "fuse_mount1" under "fuse_mount2"
$ cp -n -a -r ./fuse_mount1 ./fuse_mount2
$ # Unmount the two fuse filesystems
$ fusermount -u fuse_mount1/
$ fusermount -u fuse_mount2/
Which needed around 10G memory of my test machine's RAM of 20G ... Which is not something an average consumer machine can handle.
Bottom line, fuse-zip can use/need huge amounts of memory in some situations and if your RAM memory is limited or even average then doing huge/massive file operation such as copying is not recommended and might not work due to it exhausting system memory.
I did use the debug option i.e. fuse-zip -d … mounting the archives from other terminals and noticed excessive logging on the target mount for cp that appeared to print a lot of memory addresses which indicated big memory usage on the target i.e. fuse_mount2 while that appeared to be much less on the source mount … I haven’t had time to deeply look into it, but you can do that and investigate more with debugging enabled … So, it appears copying from is less memory demanding than copying to a fuse-zip mount.
And this is actually stated on the fuse-zip project page:
Added/changed files resides into memory until you unmount file system.
Adding/unpacking very big files(more than one half of available memory) may cause your system swapping. It is a good idea to use
zip/unzip in that case :)
And that explains the whole thing.
Therefor, for such operations, extract the archives, do whatever you want with them and then compress them again.
If, however, you must do it your way anyway ... Then, copy some files while monitoring your memory usage and when your memory gets near to full, stop copying and unmount the target fuse-zip mount wait for it to finish processing, then mount it again to copy another batch of files and so on ... However, I don't know of a tool that will let you do that automatically ... So, I wrote this bash function(fault proofing is minimal and it's meant to be a template):
fzcp ()
{
[ -z "$1" ] || [ -z "$2" ] && echo "Insufficient arguments" >&2 && return 1
source="${1%/}"
target="${2%/}"
read -r -p "What is the full pathe to the target fuse-zip mount point? : " mount_point
read -r -p 'What is the full path(including filename) to the ZIP archive mounted on it? : ' zip_file
[ -z "$mount_point" ] || [ -z "$zip_file" ] && echo "Insufficient arguments" >&2 && return 1
total_m=$(free | awk '/Mem:/{print $2}')
total_m_percent=$(("$total_m" / 100))
shopt -s globstar
for f in "$source"/**/*
do
[ -f "$f" ] || continue
free_m=$(free | awk '/Mem:/{print $7}')
free_m_percent=$(("$free_m" / "$total_m_percent"))
if [ "$free_m_percent" -lt 20 ]
then
echo "Unmounting $mount_point ..."
fusermount -u "$mount_point" || break
while pkill -0 -f "${zip_file##*/}"
do
printf "%s\r" "$(date) :: Waiting for fuse-zip to finish adding copied files to $zip_file ..."
sleep 1
done
echo
echo "Mounting $zip_file on $mount_point ..."
fuse-zip "$zip_file" "$mount_point" || break
fi
cp -n -v --preserve=all --parents -- "$f" "$target"
done
shopt -u globstar
}
That works like cp(However doesn't accept options) but for copying to target fuse-zip mount points ... It takes two arguments and used like so:
fzcp ./fuse_mount1/source_directory /home/user/fuse_mount2/target_directory
It will then ask you to provide the path to the target fuse-zip mount and the path to the ZIP file that is mounted at it … Then it will do the job … Test it first to get used to how it works before you rely on it for copying important data … The parent directories that will be created on the target depend on the source directory path that you provide so running the function from the very first/closest parent to your source directory and providing the path in as short as ./source_dorectory should give you only the tree of this directory and its sub-directories excluding real full system paths up to its root directory i.e. /.
