Amazon Glacier

For technical support visit http://support.2brightsparks.com/
User avatar
mickyj
2BrightSparks Staff
2BrightSparks Staff
Posts: 906
Joined: Mon Jan 05, 2004 6:51 pm

Amazon Glacier

Postby mickyj » Fri Sep 07, 2012 9:54 am

Recently Amazon introduced a new cloud service called Glacier. The major benefit of Glacier is that it is a very inexpensive way to store large amounts of data in the cloud.

We have had many people contact us to ask if we are going to add support for Glacier, often with the comment that Glacier is probably very similar to the existing Amazon S3 service, and presuming it will be an easy thing to do.

Yes, we are looking at Glacier, but no, it is not similar to S3. Let me explain why.

First off, Glacier has clearly been designed primarily for archiving and not backup. What is the difference between archiving and backup? The main difference is that a backup usually uses active data, i.e. files that are changing. Archiving is for securely storing static files (i.e. they do not change) for potentially long periods of time (think years or decades). In Glacier terminology files uploaded to it are referred to as archives, not files or objects.

In Glacier you cannot modify, move, or copy any file (archive) stored within it. Once you upload something to Glacier it can only be downloaded or deleted. That means the archive is immutable.

When data is uploaded to Glacier (as an archive) it is not given a filename or stored in a virtual file-system, i.e. there is no path. Once an archive has been uploaded you are given a unique ID which is just a very long string of letters and numbers that has no meaning at all except as the key to retrieving that archive in future. So Glacier cannot be used to host web sites as per S3, for example. That also means you cannot access the archives using a browser.

As Glacier uses a bunch of ID’s, it is not practical for different applications to store and retrieve archives because there is no standard way to say what the ID refers to, e.g. the filename. So you could download an archive from Glacier but all you’ll have is a meaningless ID. There is a way to store a description with an archive in Glacier, but that description is just free-form text and so could be anything at all. The application that uploaded the archive decides what it means.

Now to the most significant difference: if you want to retrieve an archive from Glacier it can take hours to get it. Yes, hours. It’s actually an asynchronous process, i.e. the application must ask for the archive (using its unique archive ID) and then it can go and do something else. Eventually a message is sent from Glacier telling the application that the archive can now be downloaded. This very slow process also includes getting a list of the archives available. That means unless the application caches all the archive ID’s locally then simply getting a list of files can also take hours.

Glacier is perfect for archiving. For example, everyone has lots and lots of files that they rarely use but need to keep a copy of (legal documents, personal photos, etc.). With Glacier you could move those files off your computer or server and into Glacier. You then have a safe off-site archive of those files. Amazon states:

Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive. The service redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives. Glacier performs regular, systematic data integrity checks and is built to be automatically self-healing.


Thanks for reading, and I hope you’re still awake! Now we want to hear from you.

What would you use Glacier to store? How would you use it? What sort of functionality would you expect an application using Glacier to support? Add your comments and thoughts to this thread, or email archive [at] 2brightsparks.com

c141heaven
Newbie
Newbie
Posts: 3
Joined: Mon Sep 10, 2012 6:15 pm

Re: Amazon Glacier

Postby c141heaven » Mon Sep 10, 2012 6:29 pm

Well, archiving is exactly what I need to do. I have thousands of photos going back 20 years or more, and other data as well that I just want to get in a safe place off site. Frankly the Glacier retrieval costs scare the @#@ out of me, but I figure if I ever got to the point where I needed to get at something (due to a fire or theft of my local copies) that cost and time would be a minor issue. I don't view it as a 'hotsite' kind of storage .. just ... reliable, long term and cheap storage.

So, all I really need to do with SyncBack is really the backup to Glacier. Basically, I'd like to have it upload the files as fast as possible, reset my archive bit, and skip that file next time. I'm presently using another product called FASTGLACIER (see fastglacier.com) It was only $30US and seems to do a pretty good job but thus far at least, they haven't given it the ability to copy only changed files (e.g. by tracking in a local database or with an archive bit or whatever).

Perhaps you might consider a separate product just for this, but it would be really nice to have SB fire off at night when it does my local backups, and send what's changed to Glacier as part of the same process.

mgnavarr
Newbie
Newbie
Posts: 1
Joined: Sun Sep 16, 2012 4:04 am

Re: Amazon Glacier

Postby mgnavarr » Sun Sep 16, 2012 4:41 am

Hi!

Good timing. I have been thinking about archiving solutions to use here at home. Here are some thoughts for you to consider...

I use SyncBackPro to backup many files. A large percent of the files are home videos, family pictures, and music.

The videos and family pictures almost never change, except to add new files. They are valuable to our family for sentimental reasons.

The bulk of the music collection rarely changes too. New music is added and metadata get updated, but most of it is static. I have spent a lot of time organizing the music so it would be unfortunate to lose all that work.

I DO trust 2BrightSparks to do a great job backing up my files. I do NOT trust Amazon, Google, or similar companies to safely backup or archive my data. At least not for a reasonable price.

I think it would be great if 2BrightSparks would integrate a home archive solution into SyncBackPro, or even separate product. The trick is to be disciplined the take a copy off-site. :oops:

Since 2BrightSparks is trustworthy, maybe they could offer their own cloud based archiving solution for small business and families.

mgn

digitalknave
Newbie
Newbie
Posts: 6
Joined: Wed Sep 05, 2012 3:43 pm

Re: Amazon Glacier

Postby digitalknave » Thu Oct 11, 2012 6:35 am

I'm with C141Heaven on this.

I've looked at S3 before and know you support it but as I'm not really planning on pulling files back unless there's a disaster I've held off. Glacier seems perfect for my needs though.

I'm a photographer and use SB Pro to back up all my photos to a couple of locations but I'm only really protecting myself against hard disk failure. If my home burnt down I'd lose stuff as its not stored offsite. Glacier seems a really simple way to do this and as mentioned above, I would only be using it as an offsite archive/backup.

I'm not really sure why you would view this differently to your existing S3 set-up but due to the cheaper cost of Glacier would love to see it integrated into SB.

User avatar
mickyj
2BrightSparks Staff
2BrightSparks Staff
Posts: 906
Joined: Mon Jan 05, 2004 6:51 pm

Re: Amazon Glacier

Postby mickyj » Thu Oct 11, 2012 7:39 am

Like most people, I have gigabytes of files (documents, source code, videos, photos, music, etc) that I want to be archived securely off-site. We understand exactly what you want and I can assure you we are actively working on it. I can't say anymore than that :wink:

We view it differently than S3 because Glacier is very different from S3. They are designed for different purposes. The API's are completely different and Glacier doesn't work at all like S3. Sure, they both store files off-site (in the cloud) but they do it in very different ways, have different features and abilities, and are designed for different requirements.

Thanks for your feedback.

plogan
Newbie
Newbie
Posts: 1
Joined: Sat Oct 22, 2005 10:47 pm

Re: Amazon Glacier

Postby plogan » Tue Oct 30, 2012 5:59 pm

All this is very helpful. Glacier looks terrific and the price is right, but there are two problems, from my perspective. (1) Verification: can users verify that a given file transferred 100% correctly? That's a big issue in digital photography. (2) Privacy: What privacy guarantees are in place? Files are encrypted, but if Amazon keeps a record of the key, that's no guarantee of privacy. Does it aggregate your data for commercial use? Does it retain the right to use your files even after you close your account, a la Google?

User avatar
mickyj
2BrightSparks Staff
2BrightSparks Staff
Posts: 906
Joined: Mon Jan 05, 2004 6:51 pm

Re: Amazon Glacier

Postby mickyj » Wed Oct 31, 2012 2:29 am

plogan wrote:Verification: can users verify that a given file transferred 100% correctly?

Yes, that can be guaranteed and is actually part of the API. Amazon also do integrity checks on the files to ensure they aren't corrupted while in storage.
plogan wrote:Privacy: What privacy guarantees are in place? Files are encrypted, but if Amazon keeps a record of the key, that's no guarantee of privacy. Does it aggregate your data for commercial use? Does it retain the right to use your files even after you close your account, a la Google?

I can't speak for Amazon, but I'd be very surprised indeed if they claimed any rights to your files or even tried to use them. Of course you could always upload them encrypted (separate from their own internal encryption) so that there is no way at all for them to use or access the contents of your files.

karl
Newbie
Newbie
Posts: 1
Joined: Wed Nov 07, 2012 7:32 pm

Re: Amazon Glacier

Postby karl » Wed Nov 07, 2012 8:06 pm

Hello

I am looking into using SyncBackPro and Glacier to handle our Disaster Recovery, basically replacing moving tapes offsite.

We would only use Glacier in a true Disaster Situation when our local backups kept inside a Fireproof Box inside a Safe inside a Safe were not available.

It is my understanding that any file that is deleted before 3 Months is charged a fine for early deletion equal to the fee if it stayed on the server for 3 Months.

Would this apply to the archive files that SyncBackupPro would be writing to Glacier each evening?

If Glacier requires keeping 3 Months of daily backups then the $0.01/GB/Month fee would make Glacier unaffordable. 250GB x 120 x 0.01/GB/Month would be @$300/Month

But if only 250GB of space was used each day then $2.50/Month (250GB x 0.01/GB/Month) would be very affordable and worth consideration.

Is there an affordable way to use SyncBackupPro and Glacier to handle daily backups?

Thanks

User avatar
mickyj
2BrightSparks Staff
2BrightSparks Staff
Posts: 906
Joined: Mon Jan 05, 2004 6:51 pm

Re: Amazon Glacier

Postby mickyj » Mon Nov 12, 2012 9:29 am

Hi,

karl wrote:I am looking into using SyncBackPro and Glacier to handle our Disaster Recovery, basically replacing moving tapes offsite.

Glacier is not the best thing to use for disaster recovery. Why? Because it will take hours to be able to recover from Glacier due to the way it works (see the original post above). Glacier is ideal for long term storage of archive data.

Glacier is not the best thing to use for backups of live data. For backups I'd use Amazon S3 instead (which SyncBackPro already supports). It has far quicker access and no issues about early deletion.

dgpete
Newbie
Newbie
Posts: 1
Joined: Sat Jan 19, 2008 3:02 am

Re: Amazon Glacier

Postby dgpete » Sat Dec 01, 2012 1:09 am

I for one will vote STRONGLY for SBPro to integrate BOTH S3 AND Glacier support!!!!!

I started using SyncBack Pro for making a backup to S3 as soon as it was available and will do the same when Glacier is supported by SBPro.

Regarding speed of access or recovery: back in the day, I would make a weekly tape backup on 3 rotating tapes, two of which were stored offsite. My day to day backup/recovery was on floppies then DD Floppies, then on RW-CDs then on an onsite, local NAS (HUGE for it's day.. gee, I think it was 120GB :) )

If you remember how LONG it took for a tape drive system to restore a machine (assuming that it worked at all, of course) then you wouldn't be worried about how "long" a retrieval from Glacier would take! lol That kind of recovery is your "Hail Mary" play...

The reason for my overly cautious approach (OK call me paranoid) is that I live on the South East coast of Florida, less than 1/8 mile from the Atlantic at an altitude of about four feet. So my fears and paranoia about local data loss might be a bit more worrisome than for others.

I still keep all my files on my RAID arrayed server and two 4TB RAID NAS units so that I can, if needed, access any file no matter it's age, almost instantly BUT I will keep ALL my old files in Glacier, say, everything older than 30 days and update the changed and added files monthly. Those files I will probably NEVER access but would not want to lose.

Currently I have ALL my data, back to year 0, up-to-date as of a few minutes ago, on S3 and that isn't cheap!!!!!!

To sum up my situation, I would like to have:
A.) all my data back to 1988 on my ONSITE local NAS units
B.) anything I might need to recover quickly (like the last rolling 24-36 months of data) OFFSITE on S3 and
C.) ALL of my data older than one month OFFSITE in Glacier
At that point I would feel I had as complete and trustworthy backup plan as I could at a VERY reasonable price.

Just my two cent's worth AND a plea for SBPro to support Glacier as soon as practical!

Thanks
Doug

cowelln
Enthusiastic
Enthusiastic
Posts: 12
Joined: Thu Nov 29, 2012 3:14 am

Re: Amazon Glacier

Postby cowelln » Sun Dec 09, 2012 9:28 pm

I second that! I want to use Glacier to store (amongst other things) all my RAW photos, and unedited videos. That way if I should lose everything, at least I can go back to the originals. That's true archiving - if no disaster happens, I will never need to restore them from Glacier.

As I'm here, it would also be a 'nice to have' feature to be able to specify 2 destinations - e.g. to have files backed up to a local NAS and to cloud storage in one hit.

Love the product!
Nick

Sade54
Newbie
Newbie
Posts: 1
Joined: Sat Feb 02, 2013 5:37 pm

Re: Amazon Glacier

Postby Sade54 » Sat Feb 02, 2013 5:41 pm

Hi ,

What is the current status concerning the Glacier implementation ?
I'm just waiting for it before buy SB pro ! :)

Dave Wilkins
2BrightSparks Staff
2BrightSparks Staff
Posts: 1879
Joined: Thu Jan 04, 2007 10:02 am

Re: Amazon Glacier

Postby Dave Wilkins » Sun Feb 03, 2013 3:03 pm

Still stock-piling feedback. No news yet, sorry.

billc.lange
Newbie
Newbie
Posts: 2
Joined: Tue Apr 02, 2013 3:31 pm

Re: Amazon Glacier

Postby billc.lange » Tue Apr 02, 2013 3:59 pm

I've just started to look at S3 as I recently upgraded to v6 pro.
I created a S3 account to do some testing before I decide how to use S3.
I've also been reading about Glacier.

I would also like SyncBack Pro to support Glacier storage.
I also have a strong interest in using Glacier for long term photo and video archival. I don't care how long it takes to retrieve the data (in hours) as I would most likely be keeping a local copy and would only use the Glacier for disaster recovery and true archival longer term storage.

It seems there are 2 paths to Glacier with different interfaces.
(1) Directly to Glacier where you push existing files or archives into Glacier.
Restoration is the operation of retrieving the archive back out of Glacier storage.
(2) Via S3 where you first backup to S3 and then specify a lifecycle which will then at some point triggers a transfer of the data to Glacier.
Restoration of S3 to Glacier data seems the reverse via S3 (restoring a temporary copy from the Glacier data to S3 storage, and not directly using the Glacier interface.

I'm still investigating but the bottom line is that the Glacier functionality at it's lower cost is something I want to and will one way or another take advantage of. I would hope that we SyncBack users will get some support from the SyncBack app that will make this simplier and easier.

I'd be OK with support for Glacier either via S3 (which seems more complex in that it requires a lifecycle definition), or directly via the Galcier interface where I assume I would have to build my own archive file with or without SyncBack's help and then push it to Glacier.
I think I would prefer the simpler way of having SyncBack help me archive directly to Glacier for my purposes of photo and video storage.

Thanks for listening.

User avatar
mickyj
2BrightSparks Staff
2BrightSparks Staff
Posts: 906
Joined: Mon Jan 05, 2004 6:51 pm

Re: Amazon Glacier

Postby mickyj » Wed Apr 03, 2013 7:41 am

Thanks for your feedback. The Glacier implementation has taken a backseat for the moment (but a lot of coding & design work has already been done).

Our current plans are to create a new multi-platform archiving product instead of trying to "force" it into SyncBack. The archiving model (and how Glacier works) just doesn't fit well with how SyncBack works (both functionally and internally). Also, SyncBack is very much a Windows program, and as the operating system world is changing rapidly, we'd like to branch out into supporting other platforms, e.g. OS X.

But I have to be honest and say it's some time off being completed. All our effort is currently going in V7 (top secret, don't tell anyone :wink: )


Return to “SyncBackPro (commercial)”



Who is online

Users browsing this forum: No registered users and 6 guests