Amazon Glacier - Off-site data storage for $0.01 per GB/month

[Arthur]

Now everyone can afford to backup off-site.

Source: Amazon Web Services Blog / Website: aws.amazon.com/glacier

We introduced Amazon S3 in March of 2006. S3 growth over the past 6+ years has been strong and steady, and it now stores over one trillion objects. Glacier builds on S3's reputation for durability and dependability with a new access model that was designed to be able to allow us to offer archival storage to you at an extremely low cost.

To store data in Glacier, you start by creating a named vault. You can have up to 1000 vaults per region in your AWS account. Once you have created the vault, you simply upload your data (an archive in Glacier terminology). Each archive can contain up to 40 Terabytes of data and you can use multipart uploading or AWS Import/Export to optimize the upload process. Glacier will encrypt your data using AES-256 and will store it durably in an immutable form. Glacier will acknowledge your storage request as soon as your data has been stored in multiple facilities.

Creating a vault in Amazon Glacier.
Glacier will store your data with high durability (the service is designed to provide average annual durability of 99.999999999% per archive). Behind the scenes, Glacier performs systematic data integrity checks and heals itself as necessary with no intervention on your part. There's plenty of redundancy and Glacier can sustain the concurrent loss of data in two facilities.

At this point you may be thinking that this sounds just like Amazon S3, but Amazon Glacier differs from S3 in two crucial ways.

First, S3 is optimized for rapid retrieval (generally tens to hundreds of milliseconds per request). Glacier is not (we didn't call it Glacier for nothing). With Glacier, your retrieval requests are queued up and honored at a somewhat leisurely pace. Your archive will be available for downloading in 3 to 5 hours.

Each retrieval request that you make to Glacier is a called a job. You can poll Glacier to see if your data is available, or you can ask it to send a notification to the Amazon SNS topic of your choice when the data is available. You can then access the data via HTTP GET requests, including byte range requests. The data will remain available to you for 24 hours.

Retrieval requests are priced differently, too. You can retrieve up to 5% of your average monthly storage, pro-rated daily, for free each month. Beyond that, you are charged a retrieval fee starting at $0.01 per Gigabyte (see the pricing page for details). So for data that you’ll need to retrieve in greater volume more frequently, S3 may be a more cost-effective service

[Arthur]

Just noticed this in the small print.

Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.

[plexer]

Wierd to be charged for deleting data?

Ben

[pete]

That has got me thinking. Wonder how long it'll be before it begins to creep into other backup tools.

I suspect deletions are expensive in terms of resource use, so they'd like to keep them to a minimum.

[TechMonkey]

Waiting 3-5 hours to get at your site backups when you are trying to restore may be an issue.

Looks good for non-essential long term archiving though.

[Arthur]

Wierd to be charged for deleting data?

Retrieval of data too. Why does Amazon have to make it so complicated!

According to this post they charge based on how long it takes them to retrieve the data. The hourly retrieval rate would be the amount of data you requested divided by how long it takes them to retrieve it (3.5 - 4.5 hours).

If it takes them 4 hours to retrieve your 3TB, then your peak hourly retrieval rate would be 768GB / hour (3072 GB / 4 hours). Your billable hourly retrieval rate would be 768GB - 1.28GB (3072 * .05 / 30 / 4 hours).

Total retrieval fee: 766.72 * 720 * .01 = $5,520.38 (~180x your monthly storage fee)

The pricing appears to not be optimized for retrieving all your data in one fell swoop. This particular example appears to be a worst case scenario for restoration because you haven't split up your data into multiple archives (doing so would allow you to reduce your peak hourly retrieval by spacing out your requests for each archive) and you want to restore all your data (the free 5% of your data stored doesn't help as much when you want to restore all your data). (Source)

[FN-GM]

why is it so cheap?

[Arthur]

Glacier is primarily designed for the archival of data that is rarely/never accessed.

[SimpleSi]

As Arthur says - they've seen a market for "just in case but prob never going to need" archiving and have priced it at the Dropbox pricing model "You'd have to be mad not to pay us a few quid a month for the fantastic service we offer"

And with the access times they specify - they can afford to compress it down to 32bytes controlled by a fractal algorithm

Simon

[Arthur]

Wired has a good explanation of the fees involved...

Is There a Landmine Hidden in Amazon’s Glacier? « Wired

An Amazon spokesperson says “For a single request the billable peak rate is the size of the archive, divided by four hours, minus the pro-rated 5% free tier.”

It doesn’t seem likely that this is Amazon’s intent for the pricing, but it has made some developers nervous. “If you wrote an automated script to safely pull a full archive, a simple coding mistake, pulling all data at once, would lead you to be charged up to 720 times what you should be charged,” one Hacker News commenter wrote.

[localzuk]

All of this would make me nervous if I were to consider using it as a disaster recovery storage solution (it wouldn't be suitable for a backup solution, due to the speed at which restoration is required for files).

If they could clarify the pricing with examples, that would help.

The cost for storage would be cheap, cheap, cheap. But in the event of a disaster? Say you're like us and a full backup is around 4TB.

You want to get your systems back up and running asap after a disaster so you request it all to download.

The end result would be a bill of $7360.49 to recover all your data. Would many schools be willing to pay that? Or would they prefer to pay more evenly over time, with no direct cost to recover?

[CyberNerd]

The end result would be a bill of $7360.49 to recover all your data. Would many schools be willing to pay that?

We only back up to an onsite storage device.
Having the extra security of an offsite backup sounds like a really good step forward.
I would be MORE than happy to pay a few thousand pounds on the off chance it was needed (it never has been needed in the 20 or so years that the school has had IT) - so it would certainly be cheaper than offsite tape storage.

[glennda]

The end result would be a bill of $7360.49 to recover all your data. Would many schools be willing to pay that? Or would they prefer to pay more evenly over time, with no direct cost to recover?

In the Event of DR you would just have it listed in the plan - we need to pay to get it up and running again, just as in any other factor such as new hardware etc

[CyberNerd]

In the Event of DR you would just have it listed in the plan - we need to pay to get it up and running again, just as in any other factor such as new hardware etc

I reckon it would be negligible compared to the amount we'd pay for hardware/overtime/consultancy in the event of a disaster (fire or something)

[glennda]

I reckon it would be negligible compared to the amount we'd pay for hardware/overtime/consultancy in the event of a disaster (fire or something)

Exactly - to rebuilt a server room even of the basic variety costs thousands of pounds 5k for the data is nothing especially if you compared hosting that 4TB data with somebody like databarracks etc who will do the restore for free.

And you could probably convince and Insurance company to pay for it!