Storage for 12,000 users

[indie]

I've been put in charge of rolling out a VLE for the Local Authority and we're looking to provide e-portfolio space for upto 12,000 kids.

 

It's a scary ammount of storage but I have no idea what kind of space we should be looking at per user, I suggested 1GB and got laughed at by my boss, does anyone know if BECTA has guidance on how much space to provide?

[djm968]

Its only 11.718750 terabytes!

[mrforgetful]

They should just supply every kid with two, 512Mb memory sticks.

 

One so use to store things on, one to mirror every day as their backup.

 

Would be so much cheaper and easier!

[indie]

I got laughed at because my boss thought 1GB wasn't enough. He said 10GB but I can't see how each kid would use up 10GB of space.

 

12TB isn't bad, but 120TB is getting rediculous!

[Geoff]

Have a look at the Storage Management section of FITS.

 

http://www.becta.org.uk/fits_om/documents/storage_management.pdf

 

But to answer your specific question, what does the system spec your boss handed you say?

[ThatBoringBloke]

I got laughed at because my boss thought 1GB wasn't enough. He said 10GB but I can't see how each kid would use up 10GB of space.

 

12TB isn't bad, but 120TB is getting rediculous!

 

I give all of our kids 50mb on our VLE and they never even use that!

[DMcCoy]

Just remember its storage cost plus the cost of a suitable backup system. A suitable backup system for 12TB that can complete in a resonsable time and hold multiple archives is not going to be cheap

[indie]

Realisitically I don't think we could afford a backup system for that ammount of space. I was just going to build multiple hot spares into the RAID array.

 

I'm the one doing the spec Geoff, as usual my boss is an untechnical bod.

[webman]

Why not keep it under 700MB so it can easily be burned to a CD?

[DMcCoy]

Realisitically I don't think we could afford a backup system for that ammount of space. I was just going to build multiple hot spares into the RAID array.

 

I'm the one doing the spec Geoff, as usual my boss is an untechnical bod.

 

but RAID is not backup.

 

rm -fR * will still delete everything on all disks quicker than you can shout "generic expletive!"

[ajbritton]

Sounds like a job for the = : * *S A T A B E A S T * * : =

[Geoff]

I'm the one doing the spec Geoff, as usual my boss is an untechnical bod.

 

Then you need to do a user analysis. This'll give you the information you need to work out the requirements.

[NetworkGeezer]

Coping for 12,000 users will always require a substantial investment.

Is one of the project aims that the portforlio is used in preference to normal SMB shares?

[spc-rocket]

I would look at SAN (Storage Area Network) which allows for greater flexibility with storage allocation. If you do find that more storage is require then you can upgrade the san or add more disk to increate the overall storage.

 

I guess with this you can start off with say giving each user 300-400mb space to see how much they use. If they require more then you can allocate more space and add more disks to san as i said. This will give you the chance to test out how much disk space is actually used rather than assuming how much they are going to use.

 

SANs (espcially ISCSI) are becomming cheaper and a lot of vendors offers SANs that uses iSCSI i.e. Dell, HP, SimpleSAN etc.

 

For the backup you can use autoloaders with LTO-3 Tapes which holds about 400Gb uncompressed! don't know how much you get with the compression probably 650Gb or maybe 700Gb.

 

Have a look at the Dell Autoloader don't know the model but i think it has two magazines each allowing for upto 8 catridges.

 

The educations discounts brings the prices to a reasonable level,

 

HTH,

 

Ashok.

[plexer]

Realisitically I don't think we could afford a backup system for that ammount of space. I was just going to build multiple hot spares into the RAID array.

 

I think you have to. RAID etc... is great for keeping availability due to the redundancy but it does nothing to prevent against data corruption.

 

If some rogue piece of software or something corrupts a load of their work without a backup you can't recover it.

 

Ben

[alan-d]

I'm sure someone will tell me I'm wrong but what will the kids need to store?

 

Surely most of the input by kids will be stored as text, spreadsheet or database. I can't see them having to store large graphics or movies as that would be part of the lesson set up by the teachers.

 

1GB would be more than enough for that. The larger quota would need to be where the resources are held.

 

Take 500MB from each kid for resources and you'll probably get it right. Remember you can always increase it later It would probably take a while for schools to get resources and for the system to get going anyway.

[john]

LTO3's will do 400Gb uncompressed and 800Gb compressed, assuming a 2:1 ratio

[spc-rocket]

LTO3's will do 400Gb uncompressed and 800Gb compressed, assuming a 2:1 ratio

 

Yeah that's whay i've read as well. However not sure if that ratio could be matched so realistically it should be around 600-670Gb i think.

 

Ash.

[john]

I should try it for you as I have an LTO3 drive but I think I would have to back up most of my servers onto it over the network to get 800GB of data, we have a lot of data but its well spread out but i suspect if i stuffed it all together we would have about 800gb

[indie]

I've been looking at the Dell tape backup solutions, the prices aren't too bad, obviously were not going to have 12TB of data straight away and their backup hardware is expandable so we could just add more drives as more data gets added.

[Andrew_C]

What size kids are you looking at? If they are doing GCSE or A level course work in Music (Tech), CDT, Art or similar subjects, then they will use LOTS of storage. If they are doing essays in History or English, much lower levels needed...

[plexer]

Highly unlikely you will ever get a 2:1 compression ratio.

 

Ben

[PiqueABoo]

12,000 users throwing around 1GB or more of data doesn't strike me as one of those d-i-y things. I'd want someone with relevant expertise worrying about bandwidth, backups, load balancing & redundancy (e.g. two copies of the data in different data centres).. oh and getting outages fixed in the middle of the night.

[robert.mabbutt]

Just a quick note, don't know if it is relevent and you can tell me where to go! lol But is there any room for expansion going to be built into this projection for storage needs?

 

Robert!

[broc]

You are starting to move into 'Enterprise' storage solutions with these sorts of numbers and as has been pointed out by PiqueABoo these are not for the faint hearted. You really want products engineered for high availability with built-in redundancy, fault tolerance & 24x7 engineering support & that goes for backups too. You should be looking for integrated solutions from suppliers such as DELL,HP, IBM & others who have experience of running these sorts of storage farms.

 

I would concentrate on defining what you need in terms of capacity, availability, numbers of concurrent users, anticipated data throughput, hours of operation, backup windows (or concurrent backup?), future growth potential, and leave the choice of equipment to potential suppliers as part of a tendering process.

 

To do it right will cost a lot of money, so let the suppliers take the risks, and don't be afraid of asking them to provide customer references or even loan equipment. Any decent supplier should be able to offer both.