GOAL: To maximize throughput between two hosts.
NOTES: There is a bug in ext4 that limits the speed on mdadm-based RAID arrays to ~350MiB/s. This bug does not appear to affect hardware raid controllers.
Distribution: Debian Testing
3. Network configuration
4. Disk array configured used to compensate for EXT4 bug (+72-114MiB/s performance gain)
1. Intel DP55KG (3ware 9650SE-16PML (> 500-600MiB/s sequential writes with EXT4 or XFS)
2. A Gigabyte P35-DS4 (Software RAID-5 (> 400-600MiB/s sequential writes with XFS, capped at 350-400MiB/s with EXT4)
3. 10GbE AT2 server adapters (Intel states these are PCI-e 2.0, but the BIOS shows otherwise, showing as PCI-e v1.0, 2.5GT/s)
4. A regular Cat6 cable was used between the two hosts since switches at this time are a minimum of $5k-$10k or more. From what I read, a regular Cat6 cable can go 33 meters on 10Gbps; whereas Cat 6a can run up to 100 meters.
5. I read a 10Gbps tuning paper, it stated one should use 9000 MTU when using 10GbE, otherwise it is possible that you may hit a ceiling of 3Gbps on a single connection. I did not bother testing with the default MTU (1500) and instead went right to an MTU of 9000 instead.
1. iperf - This shows what the network can push, regardless of disk or other I/O channels.
2. nfs - Using cp over NFS.
3. ftp - Using FTP to transfer a single file.
1. iperf benchmark
p34:~# iperf -c 10.0.0.254Using nload to watch the speed during the test:
Device eth3 [10.0.0.253] (1/1):2. nfs copy benchmark
When I copy a large file over NFS from Gigabyte (SW) raid to the Intel motherboard, I get what the SW raid can read at, approximately: 560MiB/s.
Here is the file: (49GB)I submitted a bug  to the Linux Kernel Mailing List (LKML) to let the ext4 developers know that EXT4 is capped at 350MiB/s when using an mdadm software RAID.
3. FTP benchmark
For the future if anyone is wondering, the only tweak for the network configuration is setting the MTU to 9000 (jumbo frames, here is my entry for /etc/network/interfaces) Intel 10GbE (PCI-e)
auto eth3Disk array configured used to compensate for EXT4 bug on mdadm raid arrays (+72MiB/s performance gain)
The option that gives the boost is 'nodelalloc,' the mount man page reveals:
Disable delayed allocation. Blocks are allocation when data is
copied from user to page cache.
/dev/md3 /r/1 ext4 defaults,noatime,nobarrier 0 1
/dev/md3 /r/1 ext4 defaults,noatime,nobarrier,nodelalloc 0 1
$ dd if=/dev/zero of=bigfile bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 38.6415 s, 278 MB/s
After:In the original testing with kernel 2.6.33, EXT4 performance usually did not peak past 350MiB/s. However, with the final test shown above at 392MiB/s, this is with the 188.8.131.52 kernel, again with the 'nodelalloc' option. This increased the speed from 278MiB/s to 392MiB/s, to which EXT4 is currently capped on mdadm raid arrays, at least at the time of this writing.