Anyone using Amazon Glacier for off-site/DR storage?

[localzuk]

I am still working through my Trust-wide backup/DR storage plan, and am looking at potentially adapting it a bit.

 

So, I've been looking at using Amazon Glacier for the DR side of it (off-site longer term storage).

 

Is anyone else using Amazon Glacier? Its pricing is ridiculously good - too good to be true in fact.

[NB9457]

I've been looking into this too, but obviously it's on tape, there's also retention/deletion policies which cost extra. Obviously downloading the data will cost money, but in a crisis this isn't really an issue.

 

Another company I've been looking at are Backblaze, very cheap on all data is on disk, so obviously access is much quicker and they will also ship your data in a disaster on HDDs via quickest delivery possible if you need it!

[localzuk]

Backblaze are US data centers though aren't they?

[NB9457]

Backblaze are US data centers though aren't they?

Yeah, still looking in to all the cloud so I'm not too sure on the policies of storing data and whatnot yet

[mavhc]

Does it matter when all your backups are encrypted and only you can decrypt them where it's stored?

 

Brings us to the philosophy of what actually is information, if I split it in 2 so no one person can read any of it, where has it gone?

 

Be extremely careful with Glacier, it's easy to get a massive bill.

 

https://news.ycombinator.com/item?id=10921365

https://news.ycombinator.com/item?id=4412886

Edited March 3, 2017 by mavhc

[NB9457]

Does it matter when all your backups are encrypted and only you can decrypt them where it's stored?

 

Brings us to the philosophy of what actually is information, if I split it in 2 so no one person can read any of it, where has it gone?

 

Be extremely careful with Glacier, it's easy to get a massive bill.

 

https://news.ycombinator.com/item?id=10921365

https://news.ycombinator.com/item?id=4412886

 

I was aware of the cost to download data, but someone priced up $22k for 3TB?! Jeez, now that is a lot of money..

[localzuk]

Does it matter when all your backups are encrypted and only you can decrypt them where it's stored?

 

The issue is the DPA/GDPR stuff, rather than anything else.

 

Be extremely careful with Glacier, it's easy to get a massive bill.

https://news.ycombinator.com/item?id=10921365

https://news.ycombinator.com/item?id=4412886

Things have changed since then. Their own example has 500GB retrieval costing $5 ish. Multiply it up to our full data store and we'd be looking at $100 to get the data into something other than Glacier.

[mavhc]

The issue is the DPA/GDPR stuff, rather than anything else.

 

 

Things have changed since then. Their own example has 500GB retrieval costing $5 ish. Multiply it up to our full data store and we'd be looking at $100 to get the data into something other than Glacier.

 

Ah, my calculator for the following is out of date then:

 

So 10TBs?

For ireland:

$486 storage per year

 

$4200 to retrieve it all in 24 hours, $20000 to retrieve it in 4 hours, $2500 for 48 hours

 

I'd argue that if it's well encrypted, it's not personal information any more, so where it's sent doesn't matter

Edited March 3, 2017 by mavhc
old info

[Arthur]

we'd be looking at $100 to get the data

How many weeks/months would that take?

[mavhc]

I'm read the glacier pages, still no idea, and their dev docs haven't been updated.

[localzuk]

I'd argue that if it's well encrypted, it's not personal information any more, so where it's sent doesn't matter

 

The law disagrees with you on that I'm afraid.

How many weeks/months would that take?

A DR situation would very much depend on the connection! There's about a 3-5 hour time until the data is made available, and then its available to download for 24 hours. I'd probably download it over to Azure and run it directly in there in a DR situation (as the backups are Hyper-V backups from Veeam). However, that aspect would take some figuring out before documenting as our DR policy/procedures.

 

Ah, my calculator for the following is out of date then:

 

So 10TBs?

 

For ireland:

 

$486 storage per year

 

$4200 to retrieve it all in 24 hours, $20000 to retrieve it in 4 hours, $2500 for 48 hours

 

They're very out of date.

 

This is their standard example - the slow restore:

 

Standard retrievals are priced at a flat rate of $0.01 per GB and $0.05 per 1,000 requests. For example, retrieving 500 archives that are 1 GB each would cost 500GB x $0.01 + 500 x $0.05/1,000 = $5.025

 

Scale that up to 10TB, and you're looking at $100 to download it. Expedited would come to around $3200, to have that data available in 1-5 minutes, obviously with download taking however long it takes.

Edited March 3, 2017 by localzuk

[jtotheb]

Is anyone else using Amazon Glacier? Its pricing is ridiculously good - too good to be true in fact.

 

Yes.

 

The storage costs are very reasonable because of the two fairly large caveats which you're already aware of:

1. There's no instant access to the backups. If you want to restore from them you need to wait until they're available. In testing situations we've found this preparation phase to be around 4hrs.

2. You pay a fair chunk to get the data back, especially if you're in a hurry. However, if your on-site live data and backups have been destroyed then you're probably at the point where you'll pay anything to get the data back.

 

For long-term, archival type situations, it's great. With Veeam we use Cloudberry to manage the uploads automatically. If you're not grandfathered in with Veeam for the free copy of Cloudberry then it's fairly cheap.

[mavhc]

The law disagrees with you on that I'm afraid.

 

I've read the DPA, and various guidance for schools, and nothing really mentions foreign backups that are encrypted, and the key isn't sent abroad.

 

A DR situation would very much depend on the connection! There's about a 3-5 hour time until the data is made available, and then its available to download for 24 hours. I'd probably download it over to Azure and run it directly in there in a DR situation (as the backups are Hyper-V backups from Veeam). However, that aspect would take some figuring out before documenting as our DR policy/procedures.

 

Scale that up to 10TB, and you're looking at $100 to download it. Expedited would come to around $3200, to have that data available in 1-5 minutes, obviously with download taking however long it takes.

 

Is the slow restore limited to your 10% of storage thing still? Although in that case you'd then go to the bulk pricing.

 

The tricky part would be the backup software that would work out which incremental backups could be deleted, and arrange to create glacier archives suitable for that

[Arthur]

You pay a fair chunk to get the data back, especially if you're in a hurry.

I suppose even if you are in a hurry there isn't any point in requesting more data than you can actually download with your internet connection.

 

For example the absolute maximum you can theoretically download in four hours at 100Mb/s is 180GB so requesting all 10TB in one go would be pointless.

[mavhc]

(10 terabytes) / (100 Mbps) = 9.25925926 days

 

You'd definitely need Gigabit so it's only 1 day, or a method of restoring servers/databases, then critical work files, then photos/audio/video/disk images

[localzuk]

I've read the DPA, and various guidance for schools, and nothing really mentions foreign backups that are encrypted, and the key isn't sent abroad.

 

You are still sending personal data outside the EU, which is not allowed, period, encrypted or not. Unless there are specific things in place, which with the state of the new EU-US Privacy Shield now, is hard to do.

Is the slow restore limited to your 10% of storage thing still? Although in that case you'd then go to the bulk pricing.

 

You choose which retrieval type you want, so its up to you IIRC. Bulk is cheaper than the standard method I quoted above.

The tricky part would be the backup software that would work out which incremental backups could be deleted, and arrange to create glacier archives suitable for that

There would be some configuration needed for this yes. For example, if you delete something within 3 months of uploading it, you will be charged. After 3 months its free. Veeam, Cloudberry etc... all can do stuff like this IIRC.

[localzuk]

I suppose even if you are in a hurry there isn't any point in requesting more data than you can actually download with your internet connection.

For example the absolute maximum you can theoretically download in four hours at 100Mb/s is 180GB so requesting all 10TB in one go would be pointless.

In a DR situation, you'd likely be restoring to cloud anyway, so your connection speeds will be a lot higher. You'd then progressively download things from the cloud to your new hardware as you get it in.

[mavhc]

You are still sending personal data outside the EU, which is not allowed, period, encrypted or not. Unless there are specific things in place, which with the state of the new EU-US Privacy Shield now, is hard to do.

 

That's my point though, I'm not "The Act's definition of "personal data" covers any data that can be used to identify a living individual.", it can't, because there's no key to decrypt it.

 

If I'm uploading personal data to Google Apps, google can read it/process it, so it's covered. If I'm uploading encrypted data they can't read it, so I'd argue it's not.

 

I wonder if Azure's cool storage would be cheaper if you have to import it to Azure VMs in the end.

[localzuk]

I wonder if Azure's cool storage would be cheaper if you have to import it to Azure VMs in the end.

It may, it depends on what is supported by the backup software though to be honest!

 

Just had a look - it is about double the price that I can see.

[ZeroHour]

This is a very useful tool for testing costs: https://calculator.s3.amazonaws.com/index.html

I use it all the time, it can be out of date or missing some services but Amazon usually update it quite quickly.

Edited March 3, 2017 by ZeroHour

[localzuk]

Didn't realise this but Veeam's Cloud Connect tool requires you to use a middle man. So, you can't backup straight to Azure or Amazon etc... You have to use one of these guys https://www.veeam.com/find-a-veeam-cloud-provider.html.

 

All that does is increase the cost, which is dumb.

[Arthur]

Didn't realise this but Veeam's Cloud Connect tool requires you to use a middleman. So, you can't backup straight to Azure or Amazon etc...

Could you use a program like StableBit CloudDrive to make the Azure storage appear to Windows as a physical drive and then create a new backup repository on that drive for Veeam to backup to?

 

All that does is increase the cost, which is dumb.

Agreed. I am not sure what Veeam were thinking.

[localzuk]

Could you use a program like StableBit CloudDrive to make the Azure storage appear to Windows as a physical drive and then create a new backup repository on that drive for Veeam to backup to?

 

That's an interesting idea. I may have a play with that tomorrow! My main concern is that I want the remote repo to be "offline", so it can't be affected by Ransomware. I'm playing with CloudBerry at the moment, as that doesn't mount anything locally I don't think.

[NB9457]

That's an interesting idea. I may have a play with that tomorrow! My main concern is that I want the remote repo to be "offline", so it can't be affected by Ransomware. I'm playing with CloudBerry at the moment, as that doesn't mount anything locally I don't think.

 

Be interested to hear how you get on, we're in the same boat pretty much, we want something with as up to date files as possible, but be isolated in the sense ransomware or other nasties can't affect it. So far it seems seeing as we have a tape infrastructure which is kept offsite anyway, Amazon glacier probably doesn't meet our needs, so looking more into Azure, but at the moment, having weekly tapes and monthly archived tapes seems best.

[Arthur]

My main concern is that I want the remote repo to be "offline", so it can't be affected by ransomware. I'm playing with CloudBerry at the moment, as that doesn't mount anything locally I don't think.

CloudBerry does look like the better solution.

 

Be interested to hear how you get on

Ditto.

Edited March 8, 2017 by Arthur