AWS DataSync Helps Transferring Information Between AWS, Google Cloud and Azure

Amazon recently announced that AWS DataSync now supports Google Cloud Storage and Azure Files storage as storage locations. The two new options of the data service helps moving data both into and out of AWS but data transfer fees still might still be a limitation.

DataSync can copy and synchronize data across different storage locations and move it to AWS, supporting multi-cloud workflows or data retention requirements. Relying on a proprietary network protocol, DataSync runs and verifies one-time and periodic data transfers and scales according to the size of the job. The service provides include/exclude filters, bandwidth throttling controls and supports automatic recoveries from temporary network issues. Danilo Poccia, chief evangelist EMEA at AWS, explains:

In this way, you can simplify your data processing or storage consolidation tasks. This also helps if you need to import, share, and exchange data with customers, vendors, or partners who use Google Cloud Storage or Microsoft Azure Files. DataSync provides end-to-end security, including encryption and integrity validation.

Google Cloud Storage and Azure Files storage are the first sources targeted to competitor services but they are not the only storage locations supported: AWS DataSync can synchronize data with NFS, SMB, Hadoop Distributed File Systems (HDFS), self-managed object storage and different AWS services for example Amazon S3 or Amazon FSx.

Source: https://aws.amazon.com/blogs/storage/how-to-move-data-from-azure-files-smb-shares-to-aws-using-aws-datasync/

While Poccia explains how to move data from Google Cloud Storage to Amazon S3, Rodney Underkoffler and Aidan Keane, senior specialist solutions architects at AWS, show how to move data from Azure Files SMB shares. In the past the cloud provider explained how to transfer data between AWS accounts using DataSync. Corey Quinn in his newsletter writes:

This might be one of the most impressive AWS features I’ve seen in a while. It acknowledges both the existence of other clouds as well as the fact that customers might want data to live there sometimes. Well done.

Larry Hau, director of product at Rackspace Technology, agrees:

This seems like a huge deal (…) AWS has always locked customers in and this reverses that. Egress is still a barrier but this makes data migration much easier especially for BCP/DR.

AWS DataSync might help migrating data across the major cloud providers but data transfer fees can be a significant barrier. To mitigate the impact and reduce egress fees, AWS recommends deploying the DataSync agent in the source environment as it compresses data in flight when copying.

There are no specific DataSync costs for the new storage options but moving data to AWS, customers are subject to data transfer fees on Google Cloud and Microsoft Azure. Moving data from AWS, customers are charged for data transfer out from EC2 to the internet.

The rate at which AWS DataSync copies depends on the amount of data and network conditions. For example, for data transfer between on premises and AWS, a task can fully use a 10 Gbps network link.