High IO utilization without high read/write

Question

Our server occasionally becomes laggy when writing files. using iostat I noticed that the disk utilization is abnormally high. However, the read/write per second does not seem too high

# example output from `iostat -ydx sdb 1` Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdb 89.00 69.00 2404.00 2892.00 0.00 14.00 0.00 16.87 152.27 1734.67 320.72 27.01 41.91 6.33 100.00 

What could be the cause of this phenomenon?

Answer 1

We see 89 reads per second and 69 writes per second, 158 IOPS. That's great value for HDD (anything more than 100 would be good) and it looks like this is fast HDD (maybe, SAS, maybe, 10K RPM); some (think WD Green) only can do around 40!

Yes, rotating rust is that slow. Bear with it.