SAN Solution

[penfold_99]

HI All,

 

I was wondering if it would be possible for the collective hiv mind of edugeek to help us out with a possible implementation of a san.

 

Has anyone implemented a san in there school?

 

We looking to link the san to all our servers and ibm blade centre this will be done via fibre channel card and a fibre switch.

 

What is the best solution for running the servers all off the san or having a local disk ruuning the os and mount the san windows/linux etc?

 

Does any one have a picture so i can visualise how everything fit together?

 

Thanks for your help in advance

[zag]

Sounds a bit overkill for a school, just out of interest, what are you trying to improve?

[penfold_99]

We're trying to do a number of things

 

consolidate data as were nearing 3 tb of stuff.

virtualisation so the images need run off the san.

[RobFuller]

Will you want a full blown Fibre Attached SCSI or iSCSI?

 

Are you going to be running software like VMware’s ESX, which will give you the most benefits from centralised storage?

 

Also will you want more than one SAN replicated perhaps in a different location on site to provide redundancy?

[penfold_99]

Will you want a full blown Fibre Attached SCSI or iSCSI?

 

Are you going to be running software like VMware’s ESX, which will give you the most benefits from centralised storage?

 

Also will you want more than one SAN replicated perhaps in a different location on site to provide redundancy?

 

Full blown Fibre attached SCSI.

 

We will be running KVM Virtualisation as red hat has killed support for Xen adn the bonus it's free.

 

We will need a sas/sata split as we will use sata for storage and sas to run the vm images off.

 

We will be backing up the new san to our old nas

[bossman]

@penfold_99:

I would not go down the route of fibre channels as this is way too expensive at the moment.

We are looking at the Hitachi SMS100 which is just a sealed unit and carries a 5 year guarantee.

This is connected to 2 Gig managed switches for failover via cat6 and all servers are connected to these switches with 2 connections using cat6 cabling.

Throughput is 4Gbps and the unit will give us 6.75Tb using raid 6 will give us 6.75Tb formatted space over 12 sata discs to do what we want with.

 

All this for approx 6K. We will be moving forward with this over the next few months.

hope this has helped.

 

Bossman

[RobFuller]

Why use full blown Fibre SCSI when only using SATA drives, you would really want SAS 10K or higher to benefit.

Perhaps iSCSI over fibre would be better; like jinnantonnix said even cooper is an option to start with.

 

Would defiantly have your guest os booting from the SAN but your hosts having their own disks to boot from. ESX3i now integrates onto a chip so you don’t even need HD anymore.

Make sure you plan how many machines you’re going to virtualizes and what resources they need, make sure you have enough ram in your hosts then make sure you have plenty spare capacity as you will us it and the price does start climbing when you ask for the bigger modules.

[TronXP]

I have an IBM fibre channel SAN connected to IBM servers (not blade ) we are running Novell Netware 6.5 + Cluster services. I can try to answer some questions if you would like. Just PM me

[ICTNUT]

I have just gone for an iSCSI SAN with the following:

 

HP Blades x 3 with 2 x Intel Quad Cores, 12Gb Ram, 2 x 73Gb SAS in a c3000 Chasis

 

HP MS2000i 12 disk array (6 SAS and 6 SATA) 6.3Tb dual controllers

 

VMWare ESX Server

 

All warranties with 4 hour same day on site and all licenses for the next 3 years.

 

Total Cost: £21k

 

Fibre Channel equivalent: £36k <-- Far to much money for a school and no real performance gain in a school environment over iSCSI

 

We will be virtualising 14 servers and keeping just the 2 DC's and the Exchange server as physical servers.

 

Will be using our 2 old NAS servers (1TB Each) for backups and snapshots at the other end of the school as part of an amended DR stratergy.

[penfold_99]

The reason for going fibre to begin with is we are able to afford it.

 

We're looking to maximise speed as we will be using it heavily for virtualisation and citirix.

 

With fibre it will be completely separate from our network.

[ICTNUT]

@penfold_99: Yes I agree fibre is good in that sense however I plan to replicate the seperate network the fibre can provide via the use of VLANs which should result in the same.

[bossman]

Penfold_99:

Did look at EMCs CX3 SANs which used fibre channel disks front end and sata 10k disks backend. Utilizing VMware ESX it came to approx 45k which we cannot afford at this moment but need to implement a storage solution this year hence have gone for a cheaper solution which will cover us for the next 5 years which will lead nicely to BSF and it will no longer be my problem.

[penfold_99]

We're currently looking at the ibm ds3400 with 4 fibre cards and 3tb split 70/30 for sata/sas and it's coming in about 6-8k.

 

For virtualisation we are looking at kvm as it free which will save us a hat full of money.

[dhicks]

We will be running KVM Virtualisation as red hat has killed support for Xen

 

Eh? Drat, that could cause me problems as I've just moved all our machines to CentOS + Xen... Any more details on the above?

 

--

David Hicks

[CyberNerd]

Eh? Drat, that could cause me problems as I've just moved all our machines to CentOS + Xen... Any more details on the above?

 

--

David Hicks

 

It seems quite recent news. more info here:

Red Hat adopts KVM: what happens to Xen now? | virtualization.info

 

Seems xen getting sold off to citrix didn't sit too well for rhel, developing a rivals product...

 

What happens to Xen now? What happens to the Red Hat investment on it?

The official press announcement don't say it explicitly but the choice of words lets clearly understand that Xen is the past and KVM is the future.

It's unknown if Red Hat will continue to support Xen or what will happen to those enterprise customers that adopted RHEL 5 to use the hypervisor (mostly because Xen and KVM virtual machines are incompatible).

 

In any case it's unlikely that Red Hat will drop Xen tomorrow: the company sits in the Xen advisory board and its support policy implies that any distribution must be supported for seven years.

Red Hat may want to spend this time convincing its customers that KVM is a better virtualization engine: Virtual Machine Manager, for example, has been already re-categorized as a desktop user interface for managing virtual machines.

 

in the comments: http://kraxel.fedorapeople.org/xenner/

Edited June 24, 2008 by CyberNerd

[torledo]

i hardly think redhat going with kvm is in any way a serious problem for the xen product (open or commercial implementations), kvm has a different architecture to both xen and esx and many industry experts don't consider it an enteprise grade virtualization solution at all..

 

It's become quite obvious that the wins in virtualization are in the areas of the value-added by those trying to catchup with vmware. vmware with VI3 have such a strong grip in the enterprise virtualization market, that the only things the competitors can do is offer a disruptive technology....citrix are trying to do this using their considerable presence in the thin client space to push a complete virtual desktop solution....xenserver as a standalone product is a legitimate competitor to vmware for cost conscious department, but because it lacks the maturity of vmware as a corporate solution, it's very much a case of citrix developing add-ons for the product and pitching it at a price to appeal to those reluctant to fork out for full-blown vmware. Microsoft are strictly going after the small business market with hyper-v, it's really about being an additional feature for those looking to upgrade from 2003 rather than in any way a serious tilt at vmwared dominance. Redhat is the same, i think the move to kvm is based primariy on linux kernel developments in building in kvm...not sure how this affects support for xen, but kvm isn't a serious play either.

 

One things for sure, the new competitiveness will bring vmware enteprise prices down, citrix xenapps will mature particularly in the realms of management functionality - and with both products likely to come down in price to grab market share in smb's....who needs or wants kvm ?

[CyberNerd]

torledo,

Xen wasn't a competitor in most peoples eyes until Citrix bought it - added a gui and suddenly it's a VMware rival. There isn't much difference between what citrix offer and what redhat offered in RHEL5 other than the gui and the price. Although kvm currently lacks paravirtualisation support I suspect it will get major backing as it is fully integrated into the kernel already. All distros after 2.6.20 will have kvm support and RHEL are confident that it will quickly catch up with xen. I don't think kvm is something to be underestimated just yet.

[dhicks]

Sorry, we seem to have hijacked another thread talking about generall virtualisation! I'm not that bothered if Red Hat stop supporting Xen:

 

- CentOS 5.1 is perfectly adequate at the moment, and should be so until hardware advances so much that it no longer installs on new machines.

 

- I can move to another distribution that uses Xen - the only reason I moved from Ubuntu in the first place is that 8.04 running Xen kept on segfaulting (seriously - segfaulting? In the 21st century? What's with that?).

 

- Moving Virtual Machine technologies isn't that tricky, anyway - tar a filesystem up, untar it, sort out the kernel, away you go.

 

--

David Hicks

[sahmeepee]

Sorry to go back on topic again

 

We've got an iSCSI SAN here with 4x1Gbit connections into a Gbit switch (aggregated to provide effectively a 4gbit connection). If I'd been looking to boot off the SAN I would have thought about fibre channel more seriously (fibre and iSCSI HBA cards are similarly priced), but that's not really on the horizon right now. It runs about twice as fast as the local disks in our fastest servers do, so I'm happy with that.

 

Total cost of the SAN including the 3com Gbit switch was £6100, giving 6TB of SATA in RAID6 with 8 disk slots spare for SAS or SATA. There are cheaper chassis out there which would probably do the job acceptably, but it sounds like money is no object in your case! My selection was more at the budget end between DotHill, Infortrend (Eonstor) and AC&NC (Jetstor)...

[CyberNerd]

another hijack, scsi vs fibre chanel noob questions...

 

How does an aggregated 4gb iSCSI connection compare in bandwidth to a 4gb fibre channel with no tcp/ip overhead? faster? twice as fast? 4 times?

 

Can fibre channel and iscsi be combined on the SAN? with some machines using iscsi (for fileshare) and some using fibrechanel (for boot)

 

if iSCSI uses tcp/ip (thats the cost saving..?) then why do I need a iSCSI card?

rather than network card

[sahmeepee]

 

Can fibre channel and iscsi be combined on the SAN? with some machines using iscsi (for fileshare) and some using fibrechanel (for boot)

 

I'm not aware of any SANs with iSCSI and fibre connections on the same chassis. Maybe someone else has seen one. You could have 2 SANs, but then you would also need 2 lots of switching (fibre and ethernet) and 2 lots of connectivity in your servers (fibre and gbit ethernet on top of your existing network card) so there would likely be no cost saving over just using fibre.

 

if iSCSI uses tcp/ip (thats the cost saving..?) then why do I need a iSCSI card?

rather than network card

 

You only need an iSCSI HBA if you plan to boot from an iSCSI SAN. That's because Windows typically accesses iSCSI using drivers and config settings loaded from your windows installation... and those files are on the disk you're not yet connected to! What the HBA does is give your server access to the SAN before the start of Windows boot. As you suggest, many people don't bother with the iSCSI HBA if they are only using the SAN for storage of data after Windows bootup.

[torledo]

another hijack, scsi vs fibre chanel noob questions...

 

How does an aggregated 4gb iSCSI connection compare in bandwidth to a 4gb fibre channel with no tcp/ip overhead? faster? twice as fast? 4 times?

 

Can fibre channel and iscsi be combined on the SAN? with some machines using iscsi (for fileshare) and some using fibrechanel (for boot)

 

if iSCSI uses tcp/ip (thats the cost saving..?) then why do I need a iSCSI card?

rather than network card

 

iscsi and fibre channel would normaly represent different (physical) storage fabrics.....boot over SAN can be accomplished using either fc or iscsi, application data e.g exchange stores and sql databases can be iscsi or fc (and cetainly in the case of exchange CIFS)... as for filesharing....certainly both iSCSI and FC can be used to present block storage to servers which would in turn present that storage using the filesharing protocols to client machines. I see little benefit or need to split a single servers data storage requirements between two different SAN technologies. Although doing multi-protocol in a single chassis and setting up how and through which fabric type boot and application data is accessed is certainly a viable option.

 

i'd cetainy mix and match disk types in the SAN.... Boot LUNs could be SAS or fibre channel, high IOPS applications could use fibre channel disks and secondary storage could be SATA. This can all be done using a super expensive virtualization node which has the ability to connect to different vendor storage subsystems....high end stuff, with a price tag to match, and in can be done at the low-end with single vendor storage.

 

At our level devices do exist that do multiprotocol for the purposes of choosing the appropriate connection type. Say you had a storage controller (not mentioning any vendors here) that supports the full house of Fc, iscsi and cifs/nfs....the storage that sits behind the controlers can be carved up and allocated based on best practice to the appropriate frond-end connections (fc-iscsi-nas)...the disks and the disk shelves take their cues from the disk controlers which can connect to more than one fabric type. if you have a reduandant disk controller doing mutiprotocol using a various array of ports then perfomance is a big concern, if you have seperate storage controlers for seperate storage arrays each connecting to fc or iscsi you've got no such concerns, but if i had two fabrics i personally would assign each server to one or the other not have one server connected to both. If i wanted an exchange server to connect to a boot lun for the OS and a 'data' lun i'd settle on a singe fabric type (fc or iscsi) and have a dedicated disk shelve for mirrored boot disks for all servers that boot from SAN and seperate shelves (possibly of a different disk type) to host the application's data storage (typically a RAID 5 or RAID 10).

 

if i then added an ISA server, and i chose not to invest in costly fc hba's i'd connect the isa to my seperate iscsi SAN, and if required have it Boot from iSCSI SAN. (isa is just for exampe not saying it'd be an app i consier for SAN placement)

 

the issue of haing costly, specific ISCSI adapters doesn't factor in anymore...onboard network adapters function fine as dedicated iSCSI hba's....used to be the case you had to buy an adaptec card to do iscsi for copper or fibre...not anymore, that's why iscsi is so attractive to small orgs...onboard server network adapters, off -shelf gigabit switches, cheap arrays...and hey presto. FC still kicks butt when it comes to performance mind and FC wil always be cooler.