Force Re-Scan when Destination (i.e. drive) changes

For technical support visit http://support.2brightsparks.com/
MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Mon Jun 13, 2016 6:23 pm

Hi,

I am trying to setup a backup where a full backup (re-scan) is run when the destination drive changes, i.e. Backup A vol. label becomes Backup B (as an example). Then, differential backups are run each day until the destination changes again, at which time a full backup (re-scan) is run again, and so on. I need to do this because the person I'm setting the backup for may not consistently change drives on a Monday (i.e. tell syncbak to run a re-scan on monday and differential the rest of the week) because they may not always be in the office. In that case, I would like the differentials to continue until they DO change the drive, and then run another full backup (re-scan) at that time. Thanks

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Tue Jun 14, 2016 2:34 pm

I can see some other restrictions you might have to accept (differences from exactly what you had in mind/expected) based on how Differentials work in SB, but I won't go into detail about those right now, because...

...I think your main stumbling-block is that there is no built-in functionality in SB for remembering (storing) what was the most-recent disk used and reacting to presence of a different disk by Forcing Rescan (and now store the current disk's info / await another disk change). The comparison options you can see in the Force Rescan When settings involve you entering a static disk ID (Label, Serial or HWSerial) into the profile, and telling it to act (or not) if that disk is present on the system (or not). Example: if you had a set of seven disks, you could set it to Force Rescan every time the disk with the Label 'FULL' (or 'MONDAY') was found when the profile ran, but only if that were true, and not if the inserted disk was labelled 'TUESDAY', 'FRIDAY' (etcetera) or, indeed, labelled 'SOMETHING ELSE' (the only deciding factor would be the presence/absence on the system of a drive called 'FULL' / 'MONDAY'). The only way in the scenario you have described ('two disks, rotating approx weekly') that you could leverage a disk-based value is by setting it to compare/match (or not) whether a disk labelled '...A' is present, and if so, Force Rescan. That would work once, but after the Rescan you or your user would then need to edit the profile (in respect of 'value to detect') so that next week it would react to drive '...B' (if/when inserted) instead. And so on ad infinitum. This makes no sense, as it would be simpler (and less prone to error/typos) to just 'open' the profile before running it and click the Force Rescan button when you wanted one next run.

You may be able to write an installable script (think 'plug-in') that (once you install it and configure that profile to use it) will (on first use) store an ID (in the profile's INI file) of the drive being used right then, and to compare that ID with same parameter of 'drive currently present' next run, and if/when different, Force Rescan (and replace stored-ID in the INI file with that new disk ID, ready to detect the next change - and so on). You'd need to figure out what combination of IDs to use to tell the script which drive you are interested in comparing another parameter of (with the value stored in the profile), given that you propose a routine that would (at the moment) change at least two values (Label and Serial) on a weekly-ish basis. This tends to make your 'obvious' choice simply telling SB 'which drive letter' to check other parameters of, but you might want to consider whether you can rely on the same drive letter always being allocated by Windows. Check out the Help on the Simple settings page re Alternatives (also the Help on Variables > Drives, Files & Folders) and remember that (a) yes, you can lock a drive letter in Windows to a specific device but not to two or more different devices & (b) it might only take someone plugging in a flash drive to that PC to affect which drive-letter Windows 'always' allocates to your external drive next time you swap them, badly upsetting the apple-cart. You may want to consider using %LABEL=XXXXXXXXXX% syntax in your Destination (and giving both drives the same 'XXXXXXXXXX' Label, so the profile will work with either of them plugged in). You could then include logic in your script to figure out the true currently-allocated drive letter from the Label, and from the drive-letter you can then look up the drive/s Serial/s (which will be reliably different but constant unless you re-format the disk/s). Then store/use the Serial for comparison next time, Forcing Rescan programatically if it changes (and update stored ID to 'current value').

OR, think of a different scheme. Do you really want Differentials? Would running a simple Mirror with Versioning (say, up to 10 Versions, but only if/when necessary) suit you better? That would still capture changes (including deletions) without needing 'Rescan (or not) days'. You could use the 'rotated drives with same Label' trick here also, but the Versioning housekeeping will be done 'on the fly' each run based on the actual contents of the drive being used (no need to rely on a Fast Backup database that must be rebuilt whenever you change drives). Another factor to think about is that Differentials might make up to 6 identical copies (possibly more, if disk-change is late the following week) of a file modified immediately after the last Full backup, but never edited again - whereas Versioning would make just one extra copy (of the replaced-original). This 'disk economy' could be a factor worth considering if your backup drives aren't considerably bigger than your file-set. Note that V7's Rollback facility (see menu options in Differences window / contextual Help) makes a mass Restore to a specified mid-point much easier, arguably deprecating one of the (until recently) unique advantages of Differential backups (Restore to nearest mid-point in two stages). Using a Rollback in a Versioning profile can do it in one stage...

Finally, a simple Mirror with Versioning wouldn't need to be based on a fixed cycle (nor need special arrangements to accommodate deviations from it). It would simply read the Source, read the drive you point it at (every time), compare the two, process the differences (immediately bringing the full backup [on whichever drive is connected] up-to-date, with no user intervention required) and Versioning (up to the limit you set) copies of anything it replaces or deletes.

I'll leave that with you to mull over. Good luck, whatever you decide. I think I might need a holiday :)

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Tue Jun 14, 2016 2:51 pm

cliffhanger wrote:Note that V7's Rollback facility (see menu options in Differences window / contextual Help) makes a mass Restore to a specified mid-point much easier, arguably deprecating one of the (until recently) unique advantages of Differential backups (Restore to nearest mid-point in two stages). Using a Rollback in a Versioning profile can do it in one stage...

Not to mention that for a Mirror with Versioning, a Restore to 'most-recent' is the default, needing one pass only (keeping it simple when people are using the term 'disaster' often helps). Whereas (unless the disaster happens immediately after a Full backup), a Differential Restore needs two stages, cool head, etc..

MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Tue Jun 14, 2016 8:25 pm

Wow... That was an awfully insightful and thought out response... Much thanks for that, and for many things to ponder. I"m relatively new to SB, haven't used it a whole lot - a friend recommended it.

Perhaps my understanding of some of the inner workings aren't as complete as I thought.. The manual does a pretty good job of explaining functionality, but it doesn't really (I can't find any references) explicitly explain how it actually keeps a DATABASE of files when using the Fastback feature. Currently the backup runs 7 days a week, at 7PM each night, as a differential NOT using the archive bit. On Sunday's, a full rescan is forced, backing up to the external drive in a folder called \DATA\%DAYOFWEEKNAME%-FULL\, and the rest of the week it goes into \DATA\%DAYOFWEEKNAME%-DIFF\

I guess what I'm unsure about is the database itself that determines what files have been added/modified since the last rescan. Is this database wiped out upon a full rescan, and then referenced daily until another rescan is run? Are there more than one database maintained? Are they based on profile name? Are they based on the drive ID and/or Drive label of the destination? Is there somewhere I can get information regarding this part of the program? Your help has been invaluable, and I would very much like to get more details on this stuff. Thanks again, and hope you can steer me in the right direction!

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Wed Jun 15, 2016 1:40 pm

I guess 2BS decided details of inner workings weren't necessary (in the same way that you don't need to understand how synchromesh or sun & planet gears work to drive a car). Plus, the more detail provided, the more there is to monitor / keep up-to-date. So, I'm in the same boat as you, but I may have assimilated some background over the years. Things I'm pretty sure about are commented on below; if not mentioned, I don't know or would be speculating.

  • The 'database' for a vanilla Fast Backup consists of a 'set' of two TXT files, one holding data about files, one about folders. More complex Fast Backup schemes may contain multiple 'sets' (consider, for example, the setting 'Keep fast backup data based on the actual destination directory (each destination has a full backup)' and the examples at foot of that Help section that use it)
  • Yes, the FB databases for a given profile are unique to that profile by virtue of the profile-name being part of the name of the database files (set/s), plus other filename-components (GUIDs, other suffixes, etc)
  • Yes, the 'database' of a vanilla Fast Backup is - on a Rescan run - deleted & re-created (rebuilt), while re-scanning the Destination. It is then referenced on each non-rescan run. Whether it is updated each run (generally with what it just copied) depends on the configuration you opted for (for example, a Differential doesn't update the d/b, an Incremental does). Which parts of a profile that uses multiple databases (sets) are updated by a Rescan I'm frankly fuzzy on.
=====

Your just-quoted Destination-path set-up should work OK provided you always Rescan (by automated settings or manual trigger) on the same day every time. Your first post states you would like to be able to vary that latter aspect to suit 'out of office' issues. If you do, with current Destination path for Full backups, you will instantly create multiple (i.e. more than 1, in theory up to 7) Full backups, because the inclusion of %DAYOFWEEKNAME% in the optional Full path (and if you Rescan on one or more different days) means the profile will then create/use a new Full backup folder based on that Rescan day's short name (i.e. a different path = additional full backup). Follow that to the worst-case scenario (you have eventually Rescanned on every possible day of the week) and you will have 7 Full backups (or, you will run out of disk space...) of varying* degrees of redundancy and probably deep confusion over which of them you should start a Restore from if a disaster occurs. If you plan to continue using a Differential backup, I suggest you remove the %DAYOFWEEKNAME% variable from the alternate Full Backup path soonest (and rename the actual FULL folder on disk(s) to match). The alternate-path option for Full backups is designed (as per the Help) to allow you to specify a static path\folder for Full backups; for you to use it, but then insert Variables in it, defeats that object.

* depending if/when/how often you vary the Rescan day, and which day/s you vary it to in which week, it is entirely possible you could eventually have old backups in alternate paths that are weeks or months out of date (disk space permitting)

====

Or, once again, I suggest you consider a Mirror (to avoid 'orphan' build-up) with Versioning instead. Such a profile is much easier to configure, avoids the balancing acts necessary if you may need to vary the timings, allows the same or better/faster rollback capability, and (as any given disk only ever stores one copy of any one change to a file) will likely use radically less disk space than Differentials for the same Restore capability. It also allows you to safely & seamlessly use two or more rotated drives with a single profile, provided you set all the drives' Volume Labels to be the same (you can label them on their cases using Dymo, etc, to tell them apart) and configure the Destination to work out the drive letter from the LABEL. (Just make sure you don't have two or more such drives plugged in at run-time, or the profile will use the first match that Windows enumerates, which may not be the drive you want to update). You can also introduce a blank drive (with the same Volume Label) into the mix, and the profile will happily create a new full backup and (on future runs) start building its own Version collection (without needing to edit anything profile-wise). You can also 'turn down the volume' simply by changing the number (and/or age) of Versions it should keep, and next run it will delete any surplus Versions (ditto for the other/next drive you insert). You can also 'turn up the volume' if you want. No database to worry about (everything is worked out 'on the fly' from your Versioning settings versus the disk's actual contents).

Frankly, with so many advantages, I can't see why you would continue down the Differential route. So much so, I doubt I'd be very motivated to accompany you.

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Wed Jun 15, 2016 2:31 pm

cliffhanger wrote:You can also introduce a blank drive (with the same Volume Label) into the mix, and the profile will happily create a new full backup and (on future runs) start building its own Version collection (without needing to edit anything profile-wise)

Better phrasing would probably be

You can also introduce a blank drive (with the same Volume Label) into the mix, and the profile will happily create a new full backup on it, and (on future runs which address that new disk) start building that disk's own Versions collection (without needing to edit anything profile-wise)

MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Wed Jun 15, 2016 2:48 pm

Thanks again for the insight. To be honest, I don't really have any experience or understanding of the Mirror/Versioning aspects of SB... But rest assured, by the end of the day, I hope to!

Again, appreciate all the help.

MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Wed Jun 15, 2016 4:15 pm

Ok, sorry to inundate you with questions - but as I delve into versioning I do have one right now: It seems to me that you need to run the profile in order to restore a versioned file (at least that's my understanding so far). One of the things I like about SB is that unlike Windows Backup, for example, instead of creating a single file that contains ALL the files, or multiple ZIP files that contains many files, SB makes file-for-file copies. You can easily just go onto the backup drive, navigate to the location of the file, grab it, and drop it back to the source. If, on the other hand, there is a catastrophe where the stored database for the profile is corrupted or destroyed, how would you go about restoring a version of a file and/or files easily? Thanks!!

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Wed Jun 15, 2016 7:43 pm

There is no database for Versions - as I said previously, all that is done 'on the fly' - by checking for Versions on the Destination (as part of the normal scan), noting any Versions it finds and making them available as selectable alternates in the mid-point Differences window (search Help for 'restoring versions' > top hit for a full description).

A basic Versioning profile is just a collection of easily-replicated settings - assuming yours is lost/damaged (and you didn't make a backup of it - cough!), you can simply re-create one that will display available Versions (assuming you need to Restore from them in bulk), by creating a new profile the same as you did initially, i.e

  • Choose Backup or Mirror
  • Source: where your live data is (same device\path used before)
  • Dest: same device\path you used before
  • Set the profile to Enable Versioning On Destination *
  • Pick the version-storage option you used last time (so it knows where to look for existing Versions)
  • That's it (you'll need to set your preferred number/age limits (if any) eventually, if you plan to use the new profile for backup (or mirror) 'again', but you don't actually need them in place just to Restore stuff)
  • Now, run a Restore and follow the wizard's prompts :)

Or, you can browse to the Versioned files in Explorer (which must first be set to show hidden folders\files, if not set already) - where they'll be depends on the storage option you chose originally. Or do a search of your Destination for the desired filename. Find the one with the LastModifed stamp nearest to what you want and copy it where you want it. The Version will have a WhenVersioned date/time stamp encoded into the filename, which you simply remove by renaming the pasted file. (Note: don't confuse the WhenVersioned stamp with LastModified - they are not the same thing)


* do not Enable Versioning On Source - it won't do anything except force the profile to waste time looking for versions on Source it will never find; Versions are only made of changes (replacements/deletions/moves) actioned by the profile, and a Backup or Mirror never changes files on the Source, so won't make Versions there either - only on the Destination (where it effects changes)

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Wed Jun 15, 2016 7:57 pm

PS: here's an example of a Versioned copy of a file

....\$SBV$\Buffer & SafeKeep.160613171106.txt

just copy/paste it where you want, then rename pasted copy by removing the middle bit (and one of the dots) - voila!

MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Wed Jun 15, 2016 8:35 pm

Thanks for the follow-up. Been experimenting with the versioning (need to do some other scenarios), but I can see where it's much easier to setup, more efficient and faster. Just trying to decide now on how I want to do it - The client was hit with a ransomware attack some time back. The ransomeware sat dormant for a few days before launching it's encryption attack, so restoring from the previous couple days was not an option. At the time, we were doing full backups daily, and swapping the drives every other week (problem was, they were taking about 12 hours each to run). I was able to go back to the previous week (when I was pretty certain the files hadn't been attacked) and do a restore of a full back-up. They were willing (not happy, but willing) to deal with the loss of about a weeks worth of new/changed data in the face of losing everything.

With the mirror, if the same thing were to happen, virtually all the files would have been encrypted, and in turn, the original mirror would be replaced with an encrypted version. Herein lies my dilemma - I need to find a solution that isn't going to take 12 hours a night, that is going to allow for the restore of a file or two if necessary over a period of say, two weeks, and also allow for the relatively easy restore of the entire system if something like that were to happen again. I'm afraid with a mirror/versioning solution, the restore could be a lengthy, tedious, and uncertain solution. However somewhere in there is the answer, just need to figure it out!

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Thu Jun 16, 2016 12:06 pm

If defending against ransom-ware is one of your priorities, and given another infection may have the propensity to 'crawl up the cable' and actively encrypt any backups it can find too, you should maybe give some thought to ceasing trying to store multiple 'days-worths' of backups on a single device rotated once per week (read "giving malware the opportunity to encrypt all of the backups on that device in one go"), and instead move to a scheme where a device is, once it contains the most-recent backup, removed from the system ('air-gapped') ASAP, and replaced with another device for next day's (night's?) backup. It depends on how quickly you feel you would be able to tell 'something has happened' how many devices (generations) would be a good idea, but now (while the recent pain is still reasonably acute) might be a good time to convince the client to invest in more disks (suggest 'a week's worth' at a minimum).

If successful, you could (should?) drop any plan to store multiple segregated 'days-worths' of changes on a semi-static device, and could then use a simple Backup or Mirror to a daily-rotated ext drive. Your choice between Backup or Mirror depends whether you want deletions from Source be retained on Destination (Backup - potentially polluting any Restore, but permitting retrieval if deletion was accidental) or delete the orphans (Mirror). There is a 'third way' which is to use a Mirror but also enable Versioning on Dest with 'number' limited to 1, which will make a hidden copy of the orphan[*] in case you need it back.

'12 hours every run' does sound excessive, assuming it isn't the case that it takes 12 hours to create the baseline backup but much less time next time to update that (now, pre-existing) baseline backup with 'just the changes'. Otherwise it sounds like an infrastructure or topology issue. Does your set-up involve at least one side being 'remote' from the machine running Pro (a server share accessed from workhorse PC, maybe?) - or even worse, both Source & Dest are remote (ext drive is also connected to the remote machine)? If so, consider moving SB to (or installing another copy on) a machine where at least one side (Source or Dest) is 'local' (and preferably both)

Good luck with whatever you decide, anyway.


[*] it would also make 1 Version of any file it modifies, too. There's no either/or, you get both. But a default/turnkey Restore of such a profile will ignore any Versions unless you deliberately 'swap them in', so you can avoid the 'pollution' problem by default. (Plus, 1 x 'one step back' Version being available on demand can come in handy if someone accidentally deletes page-2 of a document, saves it again, but doesn't realise what happened for a while)

MajDing
Enthusiastic
Enthusiastic
Posts: 10
Joined: Tue Jan 20, 2015 5:57 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MajDing » Thu Jun 16, 2016 4:41 pm

Once again, many thanks for the reply and insight.

I tried the daily disk rotation suggestion, however they felt it overkill, and feel that if they have a previous week's backup to go to, its a risk they are willing to take. They are an electrical contractor, and for the most part believe a week's loss isn't the end of the world. Apparently after the last attack, they found the data they lost wasn't as crippling as they feared. I'll still make the suggestion again and see how it goes... Since then, they have changed the way they handle their e-mails (that's how they got it). They are wary of emails from unknown sources, and have instructed everyone not to click on any links in an e-mail. They also are much more careful in what they do with attachments. So I cross my fingers...

The 12 hour backup was because they were running a FULL backup every day. They have huge files which are mostly made up of Autocad drawings, PDF's and job site photos. I've suggested archiving data to another location, but these guys are very old school and want everything available. Hence my search for something that would A) cut down backup time and B) give them some option to go back several days if necessary due to another attack. Which is why I have differentials running now, but plan on switching to the Mirror concept with versioning. I'm planning on using two drives rotated weekly, so the mirror will essentially be a full backup each Monday it runs after being swapped out. Am I wrong about this? Still doing some testing... Then the mirror runs the rest of the week, with versioning (keeping up to 7 copies, which I'm assuming will cover the week).

Thanks for being so helpful!

cliffhanger
Expert
Expert
Posts: 752
Joined: Tue May 31, 2011 5:59 pm

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby cliffhanger » Thu Jun 16, 2016 5:59 pm

Are you under the impression your FastBackup/Differential profile creates a brand-new full backup on Rescan now? It shouldn't (unless you used a special setting you never mentioned on an entirely different settings page to erase the entire Destination first). Assuming not, what it does on a Rescan is

  • drop the old database
  • scan both sides (instead of just Source + database) i.e. Source + Full Destination
  • do a comparison
  • spot the files that are still the same and do nothing about them (doesn't re-copy them)
  • spot the files that have changed and re-copies them to (updates them on) the Destination (ditto re 'new files')
  • replicate deletions to the Full Destination if you've told it to (configured it to Mirror)
  • create & store a new database reflecting current-state
The big contrast is that the Differential profile copies the changed/new files to a set of cycling Destinations (paths, etc) whereas a vanilla Backup or Mirror always copies changes to a static Destination (ditto if also leveraging Versioning). But otherwise it's the same, thus a vanilla Backup or Mirror will, on every run >>

  • no database involved, so nothing to drop
  • scan both sides (always) i.e. Source + static Destination
  • do a comparison
  • spot the files that are still the same and do nothing about them (doesn't re-copy them)
  • spot the files that have changed and re-copy to (update on) the static Destination (ditto re 'new files')
  • replicate deletions to the static Destination if you've told it to (configured it to Mirror)
  • no database involved, so nothing to store

NOTE: there is an option for a FastBackup (designed to be used when leveraging FB to achieve Differential or Incremental backups) to flush the cycling Differential (etc) folders (to avoid build-up of old/superseded files) each time they are addressed, before re-populating them with (only) most-recent changes. This setting is not invoked on a Rescan (see clause on option's legend "...(only if not a rescan)" ) as it would force re-copying all files (which it normally doesn't do - did I say that already?)

So, if the majority of those "huge files" don't change on a day-to-day basis, they shouldn't be getting re-copied every run (just metadata* checked, to ascertain they still match). If they are getting re-copied (check the logs?) that probably needs investigation, possibly invoking technical support from 2BS and supplying them with debug information. If nothing else, all other factors apart, it will probably be thrashing the disks to death (possibly increasing chances of disk errors >> failure).

* Of course, if you're also using hashing ("slower but more reliable method of file change detection") to detect differences (instead of just LastModified stamp and Size, which are the default) it will likely take a very long time (as it says in the Help, "dramatically increase"). Your choice if so...

MarcusMM
Newbie
Newbie
Posts: 2
Joined: Sat Jul 22, 2017 8:16 am

Re: Force Re-Scan when Destination (i.e. drive) changes

Postby MarcusMM » Wed Aug 09, 2017 11:16 pm

for me is the same, 2 USB drives containing a copy of C:\ (dayly backup), changing from time to time (once a week/once a month, but not regulary for example on first of the month). Should force rescan when drive ID changes.

Do you have an example script how to do that?

In the help file IncVars is mentioned, but I didn't found any coding for that in it.


Return to “SyncBackPro (commercial)”



Who is online

Users browsing this forum: No registered users and 2 guests