Score:0

High iowait with RAID0. CPU 100%. Static data is not cached in RAM

us flag

I ask for your help. I myself can no longer see the way out.

I have a server
1x Xeon E-2236
32RAM
4 x 10TB disks (WDC WD101KRYZ-01) - collected in RAID-0.
The disks are fast. Each individually provides 250MB/s read/write. The RAID-0 array produces 900MB / s - which is quite enough to pump a 7 gigabit channel. Test conducted with hdparm -t /dev/md0

The server is designed to distribute large video files - it is a streaming video server.

When I turn on traffic to this server, everything goes well for a while, but over time, iowait grows to its maximum. The HDDs only handle 40 requests per second - which is very small. iowait is busy worker process(nginx).

1. Why? What am I doing wrong?

I tried enabling aio for multithreading in nginx, it helped but not essential.

I increased readahead = 8192, blockdev --setra 65536 - no result.

2. Why does the kernel not try to cache static data in RAM at maximum CPU load?

I checked the disks. There are no bad sectors. HDDs - great.

3. Why does atop show a high load on only one disk? (see screenshot)
4. Shouldn't RAID-0 parallel the load across all disks?

I ask for your help. Thank you!

Michael Hampton avatar
cz flag
Spinning rust has low IOPS, which is far more important than sequential read speed. You would have done much better with ten 4TB disks. And spent less money.
Grey Hill avatar
us flag
I have a second exactly the same server, with the same disks. Works perfect. The only difference is in RAID. RAID-10 on the old server. On a new RAID-0 server. RAID-0 should be faster than 10 - right?
djdomi avatar
za flag
Hdparm is never a reliable source for benchmark.And Michaels words are true. Spinning Storage is really low on iops
djdomi avatar
za flag
By the way you can use caching to get a better performance
Grey Hill avatar
us flag
Djdomi, Please, tell me in what way do you propose to do this?
Paul avatar
cn flag
Is the RAID 10 the same quantity of disks? The same quantity of disks in RAID 10 and 0 will give the same read rate (4x). I wonder if adding an SSD and making it swap would effectively increase IOPS?
Score:0
ca flag

4 x 10TB disks (WDC WD101KRYZ-01) - collected in RAID-0

First, let me warn against RAID0 unless you are really sure that losing all data in the face of a bad disk is fine. To tell the truth only scratchpad, temporary or very easily reconstructed data should ever be put on a RAID0 array.

The server is designed to distribute large video files - it is a streaming video server

When sequentially read large files, where "large" really means "larger than RAM cache", caching is ineffective: being impossible to cache all file data in RAM, each new re-read of the same file will cause previously cached data to be dropped, effectively continuously trashing your own cache.

The HDDs only handle 40 requests per second - which is very small

Unfortunately, mechanical HDD are very slow for small, random IOs: an healthy 7200 RPM HDD provides ~70 IOPS on average when tested in isolation, probably something less when put in a RAID array.

That said, a streaming service should not hammer the disks with small, random IOs. Can you provide the output of the following commands during such an high-iowait case?

  • lsblk
  • dstat -d -n -f
  • iotop -a
  • iostat -x -k 1
Score:-2
us flag

There's only one solution!

If you need good disk speed, RAID 0 and 5 should not even be considered. Personally tested)) Yes on RAID 0 and 5 you will have a lot of free space, but the speed will be very low. Under high load your server will ask for help.

I installed a software RAID 10 and my problems are gone.

The downside to RAID 10 is that it is expensive. But if you are lucky you will find a hoster with a good price for servers.

Good luck everyone!

P.S. The question is closed!

Nikita Kipriyanov avatar
za flag
**This is certainly wrong**. RAID0 is exactly for data loss at very high linear speed, due to its parallelization and lack of redundancy. Optimal state RAID5 and RAID6 should display quite high read speeds, just a little less than RAID0, because they do the same parallelization. Your problem is somewhere else, not in just RAID level, your tests are likely faulty. Please, don't misguide others.
OnkelJ avatar
in flag
I concur with @NikitaKipriyanov here - RAID0 is faster. There has to be something wrong with either the software and/or hardware for RAID10 to be faster. Most likely software/config, since you get fair speed with RAID10.
vidarlo avatar
ar flag
For a read only such as video distribution, 10 may very well be faster, as the *same data* can be read from two disks in paralell.
Nikita Kipriyanov avatar
za flag
@vidario how is that possible? I'd rather believe RAID10 will show exactly same performance as RAID0 of same number of spindles. No more, no less, because there is no reason for it to be more, no reason for it to be less. And, just a little better than RAID5 because one less block of data in each stripe than RAID0. And just a little more better thatn RAID6 because it has two less data blocks in each stripe of the same number of spindles.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.