Score:0

Ubuntu

Is there any ways for me to load files into RAM for faster calculation

khanhlpbao

1/29/23, 3:53 AM

I'm studying with Bioinformatics now. I'm working with alignment programs like BWA. I'm wonder is there any commands or directories that allow me to copy the fastq file from hard drives into RAM and perform all alignments and calculation on it, only print the output to hard drive. Or at least it can work with RAM

101

0 + 0

ram

matigo

1/29/23, 4:13 AM

Yes, you could use a RAM disk. How much RAM do you have and how large are the files that you want to work with?

khanhlpbao

1/29/23, 5:53 AM

@matigo I'm having 20gb of RAM and each fastq file is about 2-3gb at max

matigo

1/29/23, 6:36 AM

Those files aren't too large. I wouldn't be surprised if they're loaded into the file system buffers (RAM) automatically if the software is reading through them a lot. That said, you can certainly use one of the two answers below to dedicate some memory and perform some comparison tests to see if copying the files into RAM reduce the overall processing time

vanadium

1/29/23, 8:00 AM

I agree with matigo. Chances are, if you sacrifice the ram for a ramdisk, you may not see improvements because it goes to the expense of disk cache.

Score:3

Ubuntu

cmak.fr

1/29/23, 4:33 AM

you could use RAMFS to create/mount a directory into RAM then copy your files into it. There is also another way to achieve what you want with vmtouch

sudo mkdir -p /mnt/ram
sudo mount -t ramfs /mnt/ram
sudo chmod 755 /mnt/ram

cp /path/to/myFile /mnt/ram/myFile

Of course, the content of /mnt/ram will be lost when unmounting or/and rebooting

Free the temporary mounted ram files :

sudo umount /mnt/ram
sudo rmdir /mnt/ram

0 + 0

khanhlpbao

1/29/23, 5:54 AM

Thanks, I only need it temporarily for calculation, is there a command to reboot the mount after I finished calculation?

cmak.fr

1/29/23, 3:05 PM

@khanhlpbao see edited answer, i added commands for unmounting temp ram when calculation are done

Score:1

Ubuntu

C.S.Cameron

1/29/23, 5:12 AM

Mkusb Frugal Persistent Live Install with toram

mkusb: https://help.ubuntu.com/community/mkusb, creates Persistent Live Frugal installs that will boot toram.

The Frugal install can be loaded to USB or to internal disk.

I think this makes a very fast system, if you have enough RAM.

Between sessions data is saved on a persistent ext4 overlay partition. You can also add FAT32 and NTFS Data partitions.

0 + 0

Score:0

Ubuntu

Artur Meinild

1/29/23, 3:32 PM

Ubuntu (and other Linux distros) already come with a pre-mounted ramdisk in /dev/shm. The size of this ramdisk is half your RAM (so in the OP's case 10 GB).

Instead of creating yet another ramdisk, it's perfectly fine to make a folder inside /dev/shm, copy your files there, and remove the folder again afterwards.

Examples:

$ mkdir /dev/shm/myfiles
$ cp /path/to/myfiles /dev/shm/myfiles
$ do whatever /dev/shm/myfiles
$ rm -R /dev/shm/myfiles

0 + 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Is there any ways for me to load files into RAM for faster calculation

TH: มีวิธีใดบ้างที่ฉันโหลดไฟล์ลงใน RAM เพื่อการคำนวณที่เร็วขึ้น

RO: Există vreo modalitate de a încărca fișiere în RAM pentru un calcul mai rapid

RU: Есть ли способы загрузить файлы в оперативную память для более быстрого расчета?

VI: Có cách nào để tôi tải tệp vào RAM để tính toán nhanh hơn không

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.