Device Mapper SSD Cache

I have had a look at a few SSD caching solutions, and had previously settled on enhanceio as the "best" option because it works on entire block devices and is easily added/removed. I have had some issues where the presence of enhanceio actually reduces performance because it caches every read/write, so backups, single file copies, etc are all cached even though they will never be looked at again.

This led me to try our dm-cache. According to http://www.phoronix.com/scan.php?page=news_item&px=MTM4ODA dm-cache requires an explicit flush command to be issued before it will sync the metadata to disk, which as far as I know means disk writes that have not specifically been flushed may be lost, similar to the cache on physical hard disks.

The setup is as follows:

• Partition and create a SSD device, preferably in RAID (especially for write-back caching). You can have 2 ssd devices if you want to split the data and metadata, but I am not sure at what point this will give a performance gain.
• Add the SSH to an existing LVM volume group
• Create a logical volume on the SSD for the amount of cache you want
• Create a logical volume on the SSD for the metadata (1/1000th of the size of the cache data)
• Convert the cache and metadata volumes into a cache pool
• Attach the cache pool to the appropriate logical volume

The example below shows how to create the LV where is the

pvcreate &t;ssd_dev>
vgextend <vgname> <ssd_dev>
lvcreate  -L<cachesize>G|M -n <cachedevname> <vgname> <ssd_dev>
lvcreate  -L<cachemetasize>G|M -n <cachemetadevname> <vgname> <ssd_dev>
lvconvert --type cache-pool --cachemode writeback --chunksize 128k --poolmetadata <vgname>/<cachemetadevname> <vgname>/<cachedevname>
lvconvert --type cache --cachepool <vgname>/<cachedevname> <vgname>/<lvname>

I have created a perl script (attached) to provide some stats on the cached partitions. I am not convinced that the write_hit numbers are meaningful, since I believe they indicate a written block that is in the cache. I may try to contact the maintainers or do some dev myself because there may be 4 types of write stats: cache misses that required a data read to fill in the cache block, cache misses that did not require a read from the source disk because the entire block was being overwritten, cache hits that still required flushing to disk and cache hits that avoided an IO to the main disk (because short-lived data was overwritten). It may also be worth adding a delay to write-back data to allow more IO to hit the last condition.

To remove the cache from a device, the following command can be issued, and dm-cache will flush any data from the cache before removing it:

lvremove <vgname>/<cachedevname>