I have a cluster made of 5 node, the node 1 host a big file-system exported via NFS to other nodes.
The other nodes have a fast nvme disks used for scratch.
I would like to encourage users to work on scratch as much as possible, thus I would like to propose a procedure as transparent as possible to use it, whit minimal impact on the analysis pipeline.
For simplicity let suppose that the user work by asking to the resource manager (slurm) only interactive sessions.
The best way should be: when you need a file that is present on the NFS, just read from NFS, but when you write modify/create a file from a node (or read a file just created) use the nvme. It seems that OverlayFS could be used for that purpose by overlaying the scratch on the NFS.
But how to deal with the following situation?
- the user end the interactive session
- the user start an interactive session on anther node
On the new node the files crated (or modified) on the previous node are not available.
How to properly sync back the data created on the scratch disk to the NFS mount-point, so that they will be available to other nodes?
This situation is not frequent (once per day, more or less), thus an overhead of time required to end the session is admissible, as well as the need to launch a (user space) script at the end of each interactive session.
I supposte that using lustre or other distributed file-system could be the best solution, but I'm not an expert and I'm scared to use them in production, moreover the connection between nodes is not very fast (just a 10Gb ethernet).
