Optimizations for lftp on 10GbE networks

Post date: Dec 13, 2013 7:01:18 PM

While syncing some large datasets between two Linux hosts, I wanted to know if I could do it faster, there is GNU parallel with rsync that works well, there is NFS too but I wanted to use FTP for high speed data transfers. After some analysis, I determined lftp was constantly hitting 100% utilization, I contacted the developer of lftp: Alexander V. Lukyanov. We went back and forth a few times over the past 2-3 weeks and through benchmark/analysis and checking the memory allocation buffers with strace, we finally found the best way to achieve maximum performance with lftp for 10GbE links. There was an initial thread that kicked off on the lftp list [1].

The current snapshot from today (and moving forward) has better performance from a memory allocation perspective, that is located here:

http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0.20131214.tar.gz

Further, in your $HOME/.lftprc replace the default (0x10000) with:

set xfer:buffer-size 0x20000

Compile and install the new application and then test the speed again on 10GbE:

lftp client performance with 4.4.13 (was current production version at the time)

<--- 150 152837731.5 kbytes to download

`...-11e0-a3b1-806e6f6e6963.vhd' at 87779540224 (56%) 714.06M/s eta:99s

`...-11e0-a3b1-806e6f6e6963.vhd' at 120031762432 (76%) 712.21M/s eta:50s

`...-11e0-a3b1-806e6f6e6963.vhd' at 128500216832 (82%) 667.87M/s eta:39s

<--- 226 222.418 seconds (measured here), 671.06 Mbytes per second

156549891072 bytes transferred in 223 seconds (670.60M/s)

After iterating and debugging/stracing with Alexander, the final and best result with lftp was obtained in lftp-pre4.5.0.20131214.tar.gz, the final results are shown below using the increased buffer size with many fewer -EAGAIN's and increased performance. The new version was tested and the results were confirmed via e-mail to the lftp-devel mailing list [2]

RESULTS::

156505837056 bytes transferred in 159 seconds (939.49M/s)

5.86user 152.38system 2:38.88elapsed 99%CPU (0avgtext+0avgdata 3244maxresident)k

0inputs+0outputs (2major+1288minor)pagefaults 0swaps

156505837056 bytes transferred in 173 seconds (862.45M/s)

8.60user 162.37system 2:53.09elapsed 98%CPU (0avgtext+0avgdata 3388maxresident)k

0inputs+0outputs (12major+1337minor)pagefaults 0swaps

156505837056 bytes transferred in 169 seconds (885.41M/s)

6.37user 159.36system 2:48.60elapsed 98%CPU (0avgtext+0avgdata 3364maxresident)k

0inputs+0outputs (12major+1333minor)pagefaults 0swaps

This represents a 33% performance increase: 167s over 223s with the older version of lftp (4.4.13).

Thanks to Alexander to improving lftp for high speed 10GbE networks!

[1] http://comments.gmane.org/gmane.network.lftp.user/2376

[2] https://www.mail-archive.com/lftp-devel@uniyar.ac.ru/msg01840.html