Before you begin:
Like any storage device, disk configuration is a factor, including:
• Disk speed (SSD/15k/10k/7200)
• RAID Level
• Write-back cache (Hardware RAID with BBU, Linux bcache, EMC FAST cache)
• Memory for read cache
In addition for the above, dedup appliances need RAM to store the contents of the hash. For SDFS the rule is:
• (volume size / chunk size) * 25. This equated to 256MB per TB for a 128k chunk size, and 8GB per TB for a 4k chunk size.
• You also need CPU to process the data
You also need to be aware that SDFS has 2 "sizes" for the filesystem.
• The chunk store size is the amount of space that will be used on the disks by the deduplicating storage engine. This size cannot be changed after the filesystem has been created.
• The virtual size of the filesystem. This is how big the filesystem appears to be, and can be modified later by modifying the config file for a given volume if the deduplication ratio is better or worse than originally estimated.
1. Base Debian Installation
The following instructions are for a Debian 7.0 (Wheezy) installation. The procedure should not change for later releases, but caution should be taken. Some options like language and country selection are assumed to be obvious, and are not covered below.
• Boot off the install media
• If a DHCP address was found, go back and configure the network manually if required
• Only partition the boot drive at this stage
• When prompted for software to install, only select “SSH Server” and “Standard system utilities”
• After rebooting, run:
apt-get install parted fuse fuse-utils
o Leave the smart host, but make sure the DNS entry exists
• Download the sdfs package from http://www.opendedup.org/download
• Run dpkg -i sdfs-1.2.5_amd64.deb
2. Data disk/partition config
Before setting up the data partition, you should consider the following:
• Align the start of the data partition with the RAID boundary. E.g. for a 6 disk RAID5 with 32kb stripe, the partition should start on a 160kb boundary (5 data disks x 32kb). Assuming a 512 byte sector, this would make the beginning sector divisibly by 320.
A sample parted example is shown below. In this example we have aligned the partition to 1MB boundaries (2048 * 512 bytes):
root@bhccifs01-2:~# parted /dev/sdb
GNU Parted 2.3Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
(parted) unit s
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 1048576000s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
(parted) mkpart primary ext2 2048 1048573951
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 1048576000s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 2048s 1048573951s 1048571904s primary
3. Filesystem creation
• When you create the filesystem, configure the appropriate RAID settings for optimal performance. Examples are:
o Ext4: mkfs.ext4 –E stride=16,stripe-width=80 (6 disk RAID5 with 64kB chunk size OR 7 disk RAID6 with 64kB chunk size)
o XFS: mkfs.xfs –d su=64k,sw=5 –l su=320k (6 disk RAID5 with 64kB chunk size OR 7 disk RAID6 with 64kB chunk size)
The example below is being created within a VMFS, so no alignment is done:
root@bhccifs01-1:~# mkfs.ext4 -m 0 /dev/nlsas/sqlbackups
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
32768000 inodes, 131070976 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
4000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
4. Set SDFS root mount point
root@bhccifs01-2:/etc/sdfs# mkdir –p /opt/sdfs/volumes/volumename
root@bhccifs01-2:/etc/sdfs# vi /etc/fstab
/dev/sdb1 /opt/sdfs/volumesvolumename ext4 defaults,noatime 0 2
root@bhccifs01-2:/etc/sdfs# mount /opt/sdfs/volumes/volumename
5. Create Deduplicated filesystem
You need to make sure the volume name matches the directory made above
mkfs.sdfs --chunk-store-compress=false --io-chunk-size=chunksize --volume-name=volumename --volume-capacity=size_of_dedup_filesystem --chunk-store-size=max_real_space_used
6. Mount Deduplicated filesystem
mount.sdfs -v volumename -m /export/volumename/