Amazon S3 Copying All Files Every Profile Run

For technical support visit http://support.2brightsparks.com/

Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Mon Mar 05, 2012 12:55 am

First of all, I'm so glad that SyncBackPro now includes support for S3.

As the subject says, my S3 backups seem to be copying all files from S3 to my computer every time the profiles run. Not only is this unnecessary, it could be costly, too.

Am I missing something? How can I avoid this behavior?
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Mon Mar 05, 2012 2:42 pm

What does the Differences screen actually show as the reason why it is proposing to copy the files 'again'? That the files on the cloud side and local are different, or simply not there (displayed 'ghosted') on one side (presumably 'local')?

(Note: selecting a file/row in the grid will populate the lower left pane with the attributes of the file/s on one or both sides, but not all of the attributes (if different) may trigger a copy - it depends on your profile settings whether some of them will or not)
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Mon Mar 05, 2012 7:43 pm

Looking at the lower left side, it appears that these may be the reasons "why":
- The date and time on S3 is older than the local file time
- The Attributes says "Ignored"
- Local file has no Hash value (not sure this one matters or not)

Image

What settings should I change?
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Mon Mar 05, 2012 7:44 pm

That image above is too wide to fit. Here's a link to open the full image:
http://www.embracelifephoto.com/stuff/SyncBack/syncback-s3.PNG
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Tue Mar 06, 2012 8:57 am

Unless your profile is set to base decisions on the archive bit (?), it's nothing to do with the archive bit...

But I suggest you look more closely at the date/time stamps?
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Tue Mar 06, 2012 9:18 am

Sorry, mis-read (or mis-remembered) your previous post (the one before the post with the link - which was off-screen at the time), Reading it again, I can see you spotted the difference in date/time already. That's almost certainly the issue. (Put it this way, unless you turned off date/time comparison, that would trigger an action in respect of those files)

Note that the local file shows the archive bit value as 'A', suggesting that the file has perhaps been modified recently, although it's not conclusive. But the different (newer) date/time stamp suggests the same thing even more loudly...

What changed (?) the file on the local side since it was downloaded on a previous run (?) is not something we can speculate about, really. But if it re-downloads (even though the S3 copy is older) it suggests that your profile is set so that 'S3' always wins, which you may not want. You have options to change that to 'newest (or oldest) wins' if it would help (also 'largest (or smallest) wins', but that's probably not much use in this case). Bear in mind that it wouldn't stop the profile taking action to make the two files match, but it might change which one won...

But if your aim is to stop it acting at all, you either need to track down why the file is changing LastModified stamp (and stop it happening), or set the profile to ignore that property (which of course means it would likely ignore genuine modifications, too...)
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Tue Mar 06, 2012 9:36 am

PPS: there is also a setting

Copy/Delete > Advanced > 'Do not replace newer files with older files'

But that would be skirting the aspect of 'Why are the files now different?', which I think is your bigger issue...
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Tue Mar 06, 2012 9:42 am

Just occurred to me - you mentioned "S3 backups" originally - but you have 'S3' set as your Source? Are you sure that's what you want? To back up what's stored on S3 to your PC?

Or do you want it the other way round (back up PC to S3)? Because right now, you seem to have the former set up, not the latter.
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Tue Mar 06, 2012 7:36 pm

I use S3 to store and serve things like CSS, JaveScript and images to my website. I want to have a local backup of those files. So, yes, S3 is the source and my computer is the destination.

In light of this, what would you recommend? (I haven't made any changes yet and my profiles are not scheduled until I have them dialed in - if possible)

Thanks for your patience and help, Dave!
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Wed Mar 07, 2012 10:39 am

I don't know why the files on [local PC] are showing as newer than the files on S3 - do you? It looks like some process (or person) is modifying them, which - if you don't know what/who it is - I would think would be the more important question to resolve.

So, I don't really know what you expect me to suggest?

If the files are older on the S3 side (and you want them propagated to the PC, so that the S3 state is maintained & backed up - overwriting any local changes), then carry on as you are. Or, find out what is changing the files on the PC, and get it/them to stop - then carry on as you are. The former will involve a lot of traffic, yes (as lots of files on the PC get re-set to the state as of the S3 side). The latter will probably cut it to a trickle.

If the files on the PC (after modification) are the ones you want to keep, you're backing up in the wrong direction...
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Wed Mar 07, 2012 7:22 pm

I assume that the files on my computer are reflecting the date that the backup ran and put the files on my computer, as opposed to matching the date of the files as they exist on S3. If that's the case, then my local files would always have a newer date than the files on S3 and SyncBack would always see them as conflicting and re-download everything on each run.

I believe the answer to your question about who/what is changing the date of the local files is SyncBack/Windows. Is it possible for a file on S3 with a date of 2011-12-31 to be downloaded via SyncBack/Windows on 2012-03-01 and maintain the file date of 2011-12-31? I think that would resolve it, but I don't know if that's possible.
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Wed Mar 07, 2012 9:49 pm

I just did a test that did exactly that (preserved the dates/times as stored on S3 exactly - to the millisecond). I created a new profile to back up a test bucket (that already existed / was populated with a few test files) to a brand new folder on one of my PC's internal SATA drives. All files downloaded had the same dates/times as on S3. A repeat run found no differences in date/time whatsoever. So I cannot replicate any such thing.

Don't suppose your D: drive is actually a NAS, is it? Or something else other than a standard internal hard drive on your PC? That might explain a lot...
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby adamcgeorge » Wed Mar 07, 2012 10:35 pm

I don't understand how we're getting such different results. I'm on Windows 7 backing up a local, internal, SATA drive. I even tried another internal drive with a new folder just to see if it made a difference. The files downloaded from S3 to my computer always end up with a date created and date modified matching the time they were downloaded as opposed to matching the time of the files as they exist on S3.

My settings make the profile a Mirror profile, which is what I want. Is that making a difference?

Here's my profile description:
http://www.embracelifephoto.com/stuff/syncback-s3profile.PNG
adamcgeorge
Newbie
Newbie
 
Posts: 7
Joined: Tue Oct 05, 2010 4:45 pm

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Thu Mar 08, 2012 1:31 pm

Hi

Nothing to do with Mirroring (that decides 'which' files get copied/deleted, not 'how')

I'm now able to reproduce this provided I use a file for testing which I uploaded to the bucket via AWS (not by Pro - previously my test files had all been uploaded by Pro).

I think the problem is related to meta-data that is not set by the manual (or 'other') upload process (whereas Pro does set that, and relies on it), and it's thus reading that blank property when deciding what to force-set the local copy to (and I guess that because this is blank, the Windows API we use is defaulting to 'now')

I haven't fully got to the bottom of this, and will discuss with the developer, but we are in different time zones, so will need to get back to you.

In the meantime, you could probably sidestep it by (on next run/s)

* right-click in grid > Select All
* right-click selected files > Skip
* right-click in grid > Unselect All
* sort by 'date/time on S3 side' column
* identify the range of files that have changed since the time of last run (if any)

- if this is non-zero, shift-click to select that range, then right-click the new selection and pick 'Copy to PC' (or whatever your local side is labelled) - then Continue

- if this is zero, leave them all as Skip and click Continue (to get a Success in the Result column - otherwise you'll need to Abort, which will be displayed as such, which you may not want)

Appreciate this is not a solution you can Schedule, but it should enable you to get a backup manually without re-copying everything every time.
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon S3 Copying All Files Every Profile Run

Postby Dave Wilkins » Thu Mar 08, 2012 2:32 pm

Just had another idea for a workaround. Do the select-all trick as mentioned in the previous post, but then right-click > 'Use Details From PC' (or whatever your local label is)

This will set the extra meta-data Pro is expecting on the current S3 files (yes, it will set it to match your local copies, which look newer, but at least they will match thereafter).

It's not a complete cure (any new/changed files that are uploaded by other means that do not set the meta-data we do will have the same issue, and need the same treatment).

I also notice that the test file I was using now shows (in the AWS console) the matching LastMod (matching my local copy) in the meta-data (bottom pane, Meta-data tab), but the 'external' LastMod it shows in the main grid (and bottom pane 'header' on left) is now set to the date/time the attributes were copied there. Argh. But it doesn't now show up in Pro as different, which may be the main thing...
Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
 
Posts: 7789
Joined: Thu Jan 04, 2007 10:02 am

Next

Return to SyncBackPro V6/V5 (commercial)

Who is online

Users browsing this forum: No registered users and 3 guests

cron

User Control Panel

Login

Who is online

In total there are 3 users online :: 0 registered, 0 hidden and 3 guests (based on users active over the past 5 minutes)
Most users ever online was 725 on Thu Feb 20, 2014 2:37 pm

Users browsing this forum: No registered users and 3 guests