Optimizing Cloud Backups

Author: Swapna Naraharisetty, 2BrightSparks Pte. Ltd.

Cloud services allow users to upload large amounts of data at affordable prices and provide the ability to access data from anywhere in the world via a wide range of devices. Because of this, cloud storage has become a popular way for consumers and businesses to back up their files.

However, depending the size of the data to transfer and the available bandwidth over the network, the backup time may increase substantially and can affect your system performance. Therefore, optimization is an important factor to consider when using cloud storage services for backup.

SyncBackPro provides several features to optimize your backups with cloud services such as Amazon S3, Azure, Google Storage, Backblaze, OpenStack, OneDrive, Google Drive and Dropbox.

Faster cloud transfer (S3, Azure, Google Storage, Backblaze B2, OpenStack, Egnyte)

SyncBackPro supports multi-threading (multiple simultaneous connections) when scanning for changes and transferring files to/from cloud services that supports threaded access e.g. Amazon S3, Azure, Egnyte, Google Storage, Backblaze B2 and OpenStack. This feature significantly improves the scan and copy speeds and overall performance of a profile run.

By default, SyncBackPro uses a single thread when scanning for changes, and 5 threads when uploading/downloading files from cloud services that supports multithreading. However, depending on your system performance and bandwidth available, you can increase the number of scan and transfer threads to use under:

Modify profile > Expert > Cloud > Advanced > ‘Number of scanning threads to use’ and ‘Number of upload/download threads to use’ options.

However, if too many threads are used then the performance can be degraded as it overloads the network, CPU, RAM, etc. Therefore, you need to experiment to find the optimal number of threads suitable for your environment. The maximum number of scan threads supported is 100 and transfer threads is 150.

Note: The multithreaded scanning feature is available only when retrieving the contents folder by folder. Hence, you should un-tick the option ‘Retrieve a list of all the files and folders then filter’ from Cloud > Advanced settings page, before enabling multi-threaded scanning.

Parallel cloud transfer (S3, Azure, Google Storage, Backblaze B2, OpenStack, Egnyte)

SyncBackPro V9 supports parallel file transfers. The difference between this and multi-threaded file transfer (above) is that parallel file transfer is for smaller files. It works along with multi-threaded file transfers. To use this feature in your profile:

Modify profile > Expert > Cloud > Advanced > ‘Number of worker threads’

When used, SyncBackPro will upload small files (files that are too small to be transferred using multiple threads) at the same time. How many are transferred at the same time is specified in the setting above. This can have a greater benefit to the performance of your profile than multi-threaded transfers.

Delta changes list (OneDrive, Google Drive and Dropbox)

The Delta API provides the ability to quickly retrieve all file changes on the cloud server from a specified checkpoint (e.g. from a previous profile run to the current run) and minimizes the scan time of a profile run.

This option is enabled by default with cloud services that allows retrieving a delta changes list - Google Drive, OneDrive and Dropbox. However, if you experience any problems with the use of delta API (e.g. SyncBack takes longer time to retrieve delta changes, the profile run appears to be hung in scan phase) then please tick the option ‘Do not use the delta API’ under:

Modify > Expert > Cloud > Advanced settings page.

Please refer to the  SyncBackPro help file Cloud settings page for additional details on Delta API.

Transfer Acceleration (Amazon S3)

Transfer Acceleration is a feature implemented in Amazon S3 to accelerate the data transfer between a client application and an S3 bucket. It is mainly designed to maximize the speed when transferring large volumes of data (gigabytes to terabytes of data) over long distances (across countries or continents) and improves the upload/download performance.

To use transfer acceleration option when transferring files using SyncBackPro:

• First, enable transfer acceleration on your S3 bucket via AWS console. Note that it might take up to thirty minutes for the feature to become effective. For information on how to enable this feature, see the Amazon S3 documentation.


• Modify your S3 profile > Expert > Cloud > Advanced > tick the option ‘Use Amazon S3 transfer acceleration’ and run the profile.

For more information about pricing and requirements for using transfer acceleration, refer to the Amazon S3 Transfer Acceleration document.

Faster retrieval from Amazon Glacier

Glacier is designed for archiving files, because of this restoring a file from Glacier is a time-consuming process - first, you send a request to Glacier for an object, wait for 3 to 5 hours for the object to become available and then download the file. Refer to our article Amazon Glacier for details on Glacier storage or retrieval process.

However, Amazon enhanced the Glacier retrieval process by introducing two new retrieval options Expedited and Bulk (in addition to the standard retrieval process discussed above) to speed-up the data retrieval.

Expedited: This option allows you to quickly retrieve your data within 1 to 5 minutes, but you need to pay a bit more to expedite the retrieval process.

Bulk: This retrieval method is useful when you need to retrieve large amounts of data within a day. Bulk retrievals typically complete within 5–12 hours.

SyncBackPro supports Standard, Expedited and Bulk options when restoring data from Glacier. You can choose the retrieval tier to use for your S3 profile under:

Modify > Expert > Cloud > Advanced > ‘Glacier retrieval tier to use’ option.

and expedite the restore process. For additional information on retrieval options and pricing, refer to the Amazon documentation.

Fast Backup

When using SyncBackPro V10 (or newer) you can use Fast Backup with all cloud services except Backblaze B2 (Backblaze B2 must be used via the S3 compatibility interface). Fast Backup can only be used when copying files to a cloud service and those files on the cloud service are only being changed by the profile itself. See our Fast Backup article for more details.

Conclusion

Using these optimization techniques, SyncBack users can backup files to the cloud server at a much higher speed than before and make the most efficient use of their network bandwidth, cost and time.