Before posting, and to avoid disappointment, please read the following:

  • This forum is not for 2BrightSparks to provide technical support. It's primarily for users to help other users. Do not expect 2BrightSparks to answer any question posted to this forum.
  • If you find a bug in any of our software, please submit a support ticket. It does not matter if you are using our freeware, a beta version or you haven't yet purchased the software. We want to know about any and all bugs so we can fix them as soon as possible. We usually need more information and details from you to reproduce bugs and that is better done via a support ticket and not this forum.

AWS S3 File Scan Very Slow

For technical support visit https://support.2brightsparks.com/
Post Reply
Ben Amada
Newbie
Newbie
Posts: 5
Joined: Sun Nov 18, 2012 6:21 am

AWS S3 File Scan Very Slow

Post by Ben Amada »

Hello, I'm running SB Pro v8.5.33.0 v64. I setup a mirroring profile between AWS S3 and the local drive. The profile is taking hours to run, in particular the scanning portion of it. There are only a few changed files that need to be copied down.

There are about 190k files in total spread out over different subfolders. I've experimented with "Retrieve a list of all the files and folders then filter" on and off. When it's off, the scan time is 3 hrs 50 mins. When it's on, the scan time is 2 hrs 40 mins. Otherwise I'm using the default settings, the most applicable looking one I see is 5 upload/download threads. I have "Use newer Amazon S3 security" checked as I was getting errors before checking that, it's the AWS Ohio region. Use encrypted (https) connection is off. There is some physical distance between the Ohio S3 bucket and the server SyncBack is on which is in the Ireland AWS region.

Are there any suggestions on what I can do to speed up the scanning? 2 hrs 40 mins is a long time for scanning alone. It seems like pulling down meta data on a list of 190k files shouldn't take that long, granted I'm not familiar with the inner workings.

Thanks,
Ben
Swapna
Expert
Expert
Posts: 1026
Joined: Mon Apr 13, 2015 6:22 am

Re: AWS S3 File Scan Very Slow

Post by Swapna »

Hi,

Note that your version of SyncBackPro is outdated. Please download and install the latest version from the link below:

http://www.2brightsparks.com/assets/sof ... _Setup.exe

Do not uninstall and reinstall, or you'll lose your profiles & settings. Just install 'over the top'.

After upgrading, please retry the profile run. If the problem persists, submit a support zip with debug logs of the scan phase to [email protected] for further investigation. To do this, enable 'Output debug information' from Preferences menu and run the profile in manual mode (ctrl + R). When the Differences Window is displayed abort the profile run and submit the support zip as per the instructions given here:

http://support.2brightsparks.com/knowle ... les/213970

If your Zip is over 30MB, contact us for alternate instructions, advising the actual size. Do not try to attach to the ticket!

Do not post (nor links to) Support Zips on this open forum.

Thank you
Ben Amada
Newbie
Newbie
Posts: 5
Joined: Sun Nov 18, 2012 6:21 am

Re: AWS S3 File Scan Very Slow

Post by Ben Amada »

Hello Swapna and thanks for your assistance.

I upgraded to the latest version, captured debug logs and sent them to support.

Regards,
Ben
simonjshaw
Newbie
Newbie
Posts: 2
Joined: Thu Oct 04, 2018 12:37 pm

Re: AWS S3 File Scan Very Slow

Post by simonjshaw »

I'd be very interested to hear the outcome of this as I'm hoping to deploy SyncBack to our customers in conjunction with Amazon S3.
My testbench system scan is at 30 minutes for about 70,000 files using the "retrieve list then filter" option.
Using fast backup I could scan less frequently, but I too can't see why it should be this slow.
It might be particularly annoying if the restore operation were equally slow just to get back a few files... but that's yet to be tested.
Version: 8.5.90.0
Post Reply