Join Log in
Score:0
avatar Server

scp with -C flag does not perform compression

ru flag
Anish Sana

The scp man page shows that the -C flag can be used for compressing a file on the fly while performing a remote copy -

$ man scp | grep "\-C"
     -C      Compression enable.  Passes the -C flag to ssh(1) to enable compression.

However, when I perform scp with and without the flag, the file size remains the same -

Without the flag:

$ scp root@remote-host:/path/to/file/* .
Password:
core_dump                                                                100% 7832MB 110.6MB/s   01:10

$ ls -lh
total 7.7G
-rw------- 1 user group 7.7G Jan  4 16:19 core_dump

With the compression flag:

$ scp -C root@remote-host:/path/to/file/* .
Password:
core_dump                                                                100% 7832MB  69.8MB/s   01:52
$ ls -lh
total 7.7G
-rw------- 1 user group 7.7G Jan  4 16:21 core_dump

I've tried several other options but all yield the same result:

$ scp -o Compression=yes root@remote-host:/path/to/file/* .

$ scp -C -o Compression=yes root@remote-host:/path/to/file/* .

$ scp -C -o Compression=yes -o CompressionLevel=9 root@remote-host:/path/to/file/* .

Is there something that I'm missing here?

143
0 + 0
ssh
scp
Score:1
John Mahowald avatar Server
cn flag
John Mahowald

ssh -C is compression for data in motion on the wire, not data at rest. In other words, ssh compressed and decompressed in the transport layer.

With ssh and pipes, its possible to compress on the remote host, get that compressed stream as standard in, and write it out on the local host. Which compresses data in motion and at rest.

ssh root@remote-host "zstd /path/to/file/core_dump --stdout" > core_dump.zst

Quotes are significant, indicating the remote command. Replace zstd with gzip or xz for your desired compression format.

Also possible to pipe tar archives through ssh, not necessary here with a single file.

0 + 0
ru flag
Anish Sana
This makes sense. I sort of had the feeling that the data was being compressed and decompressed, mainly because it was taking longer to `scp` with the `-C` flag, but couldn’t find any concrete information on that. Thanks!
0
Reply
John Mahowald avatar
cn flag
John Mahowald
Yes. With sufficiently large files, the limiting factor is how fast ssh can decompress, or even the write speed of storage. With sending compressed streams there is no decompression step. Total bytes transferred and written is reduced by the compression ratio. And zstd is fast and multi-threaded.
1
Reply
Score:0
another victim of the mouse avatar Server
cf flag
another victim of the mouse

In ssh, it allows gzip compression (under the scp).

This will speed things up on sluggish connections; but, on any decently fast connection (100Mbit or faster), the compression will almost certainly slow things down.

It will be more or less efficient than zip depending on whether gzip (particularly gzip -6) is more or less efficient than the compression level you choose in zip.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.