I have just finished troubleshooting an issue where DRBD was causing a bottleneck for an application (ddumbfs) that has an open mmaped file. The issue was that ddumbfs issues fsync() calls periodically on its mmaped files, and when these were happening I/O would pause while the data was being replicated.
The issue is partially fixed by increasing the max-buffers and max-epoch to the maximum values, to allow the maximum number of operations to be in-flight at any given point in time:
resource _DRBDRES_ {
net {
unplug-watermark 16;
max-buffers 20000;
max-epoch-size 20000;
}
}
I tried a number of code changes to ddumbfs in order to cause the data to be flushed more frequently. Calling fsync() more often killed performance, since it looked like all I/O is blocked until the fsync finishes. Calling msync(,,MS_SYNC) was similar, and msync(,,MS_ASYNC) is a noop in Linux according to the man page, although I tried both without success.
The end result was to set /proc/sys/vm/dirty_bytes to 10-20MB (I ended up at 20971520). From whatI can tell, this causes the kernel to start flushing dirty buffers when there is 20MB or more of dirty data. On this server which is dedicated to run ddumbfs for deduplicated storage, this setting should not cause any other issues (it could reduce performance in many other scenarios)
There may be other settings in /proc/sys/vm/dirty_* that could be modified to achieve the same or similar result.
*** UPDATE ***
In the end I solved the underlying issue by modified the ddumbfs code to NOT sync the index during normal operations, and to only sync when marking the FS as clean. The logic here is that unless the filesystem is unmounted cleanly, the index is not trusted, so there is no point running the periodic syncs.