Determining the optimal protocols for transfer of large data files

Abstract

The transfer of large data files remains a necessity for scientific research, including Bioinformatics, despite alternate solutions like cloud computing. When transferring large files, there are many transfer protocols to choose from. In this paper, a lightweight testbed is developed and used to test the SCP, FTP, HPN-SCP and GridFTP protocols for transfer of large files between the Universities of Cape Town and the Western Cape. A comparison of GridFTP’s ‘UDT’ mode (built on UDP) and its default mode is also done using the same testbed. We investigate the effect of a congested network on transfers. Speed and data efficiency are compared on a stable network to determine if any one protocol is significantly better than the rest. We find that for network speeds of around 200Mbit/s, the protocols achieve very similar speeds. The TCP based protocols achieved very similar efficiency, while GridFTP in UDT mode had much higher overhead.