Rspec performance redux

I previously posted on performance tuning of my virtual machine, focusing on the speed at which Rspec operates, refer rails and rspec test performance.  I’d concluded there was little I could do.

I’ve still been a bit disappointed with the throughput – basically it was taking 2-3 seconds per test condition, which is a long time of watching dots moving across your screen.  The sample app I’m working on has only about 10 objects in it, nevertheless it was taking 30 minutes to run the full test suite.  Clearly that wasn’t going to make me happy.

I did some more work today to see what I could do to improve it, and I’ve taking the test suite down from 30+ minutes to 3 minutes.  Which I’m pretty happy with.

I did a grab bag of changes.  First was to move the whole virtual image onto raid 10 instead of raid 6.  In theory this makes writes much faster.  The logic here is that a 6 disk raid 6 array needs to write to each and every disk – the 4 data disks and the 2 checksum disks.  This means waiting for the heads to move on each disk.  That same setup in a raid 10 configuration gives 3 pairs of raid 1 disk – so any given write needs to hit only two spindles.  In short you can do 3 writes in parallel, it should be 3 times faster.  This didn’t appear to make a material difference though.

Next, I looked at the IO performance.  I could see that the virtual had very high IO wait, but in theory I’d set the VM IO strategy to writeback – which should mean that it commits to the cache on the host server, then continues on assuming it’s written.  It didn’t seem to be doing that.  I started suspecting barriers, so I made a set of changes to the mount options and other bits and pieces, following the instructions from here (and some bits I harvested elsewhere).

The changes were firstly in /etc/fstab, changes to the mount options of my ext4 filesystem:

  barrier=0,noatime,data=writeback,nobh

Then I also changed some tuning options directly on the filesystem whilst it was mounted:

  tune2fs -o journal-data-writeback /dev/vda1

I made some changes in the mysql configuration file, although I’m not convinced they were the important bits:

  innodb_flush_log_at_trx_commit = 0
  query_cache_type = 1
  query_cache_size = 10M

Overall, this resulted in an approximately 10x improvement in test throughput – which is pretty awesome.  My complete test suite runs in 3 minutes.

Of course, the annoying bit is I went to check the better half’s laptop to see how they run for her.  She has a Mac Air with SSD.  Her tests run an order of magnitude faster again.  Sigh.