Join Log in
Score:0
Ealrann avatar Server

Increase memory usage on NFS Server

kw flag
Ealrann

I'm producing data (100GB files) that are finally copied to a server through NFS v4.2, on a 10Gb network. These files are stored on many HDDs, with XFS formatting (one copy per target drive).

When the copy tasks are running:

  • There is a huge memory usage on the client (could be more than 64GB, it takes as much memory as it can).
  • But almost no ram is used on the server.

I would like to reduce the memory usage on the clients, since they're continuously producing data and it slows them down. On the opposite, the server is mainly unused.

I guess since HDD are slows on the server, the client buffers as much data as it can to make the copy less blocking. I cannot change the hardware setup.

Is there any way to force the server to cache more data ? I would prefer to prioritize using server memory rather than client memory.

The NFS configuration:

10.0.3.1:/          /mnt/field  nfs  nfsvers=4.2,noatime,nodiratime,_netdev,noauto,x-systemd.automount,x-systemd.mount-timeout=10 0 0

/etc/exports:

/mnt        10.0.0.0/16(rw,async,fsid=0,no_subtree_check,crossmnt)

Nic configuration on server:

MTU 9000
rinbuffer tx 512, rx 1024

Nic configuration on client:

MTU 9000
rinbuffer tx 1024, rx 512

Edit: As requested, /proc/meminfo:

Client ------------------------------------------ Server -------------------------------

enter image description here

A monitorix of the memory usage on this client:

enter image description here

Network usage:

enter image description here

Note: The client uses a big tmpfs (100GB) to compute. I thinks this tmpfs is never subtracted from the Available memory count.

Edit2:

The correlation between network and memory usage is more obvious on the other client (I should've started with that). This client doesn't use any tmpfs.

enter image description here

enter image description here

479
0 + 0
xfs
nfs
John Mahowald avatar
cn flag
John Mahowald
Please edit your question to add `/proc/meminfo` from both client and server hosts, under load.
0
Reply
Ealrann avatar
kw flag
Ealrann
Ok, I just added it.
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
Can you also provide a copy of your `/etc/exports` too?
0
Reply
Ealrann avatar
kw flag
Ealrann
Yes, I added it. I put it here too for simplicity: `/mnt 10.0.0.0/16(rw,async,fsid=0,no_subtree_check,crossmnt)`
0
Reply
Score:3
Matthew Ife avatar Server
jo flag
Matthew Ife

I would like to reduce the memory usage on the clients, since they're continuously producing data and it slows them down.

How do you know this? Most of the client memory is in page cache, which is entirely normal and even improving the buffering on the server wont prevent aggressive caching of this data client side.

Have you tried flushing the page cache (as a test) and seeing how your application performs without the page cache usage?

NFS has whats known as 'close-to-open' consistency, meaning the contents of data and metadata is really only guaranteed to be stable when you've not got the file actively open (that is, another client can change the file on another system and you'd be none the wiser).

Because of this consistency limitation, page cache is relied upon by the NFS client systems applications to ensure the data is available for a read back should that become necessary.

All that being said, without knowing whats going on in your /etc/exports one method to offload more data onto the server might be to ensure you mount NFS on your clients with the sync mount option and on your server export the paths with the async mount option.

This will have the effect of ensuring writes are committed to the server on the client side, whilst the server will always reply 'done' before its committed the data to the disk.

This will impact clients throughput though as you'll be causing latency due to validating every request on the client side, but the server will buffer much more data since it wont wait for the data to land on disk first. You'd also probably want to twiddle the dirty_write_centisecs and other bits on the server to allow it to buffer more data into writeback.

Heres the rub though -- this is likely to cause the clients to slow down and the reduce the servers integrity in the face of crash. If the server crashes you could lose data.

Also, this wont affect memory usage for page cache on the clients which NFS has no real control over.

All in all, I'm skeptical reducing the clients memory usage (if its the page cache you're measuring here) will improve your clients performance.

0 + 0
Ealrann avatar
kw flag
Ealrann
Thank you for your answer. I know that because the first batch is produced faster, but as soon as a copy starts, it gets ~10% slower. I think my process is limited by memory access; so I think that using memory for anything else (here network copy) is the reason for the slow down. But it's not a big deal, I just try to optimize a bit. Since I can afford data loss, I'll try your `async` <- `sync` tip.
0
Reply
Score:1
John Mahowald avatar Server
cn flag
John Mahowald

No, forcing use of less memory is likely to make things slower, not faster. You are spending power on 188 GB of fast DRAM already, might as well use it.

Client host has 188 GB MemTotal, and is using 162 GB of it for Cached. Actually quite low on the memory demands, note 123 GB MemAvailable that can be freed very quickly. Most of the 40 GB of Shmem is probably the tmpfs.

As Cached + Shmem adds up to more than MemTotal, I suspect tmpfs is being counted twice, as shared memory and in caches. Would also explain how Cached minus Shmem is approximately MemAvailable, tmpfs lacking persistent storage cannot be freed.

Server side, 15 GB and change MemTotal, 13 GB in Cached. Plenty of available memory on this host. Presumably, most of what it does it serve files, not a lot else of memory demands.

Without evidence of overhead such as heavy vmscan activity or running low on MemAvailable, I do not recommend taking action.

0 + 0
Ealrann avatar
kw flag
Ealrann
Thank you, I'll check vmscan activity.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.