SAN Solution

[CHiLL]

I'm pretty new to all this, so I'd like some advice on SANs. We are debating whether to implement a SAN into our network. W have several file servers across the site, which are nearly full of data. We'd like a central storage solution, so we can use our servers just for applications, rather than that and file shares.

 

All of our network equipment is HP, provided by Zentek, and HP do SAN solutions, so they're the ones that are drawing our eyes at the moment. However, I don't know of any decent other solutions from other companies.

 

Ideally, we'd like the chassis and a customisable amount of hard drives. I've noticed HP's solutions are including the drives as well as the chassis. Such as 7TB over 42 drives. What if we don't want to use all the 42 drives? We don't want to pay for them all. Having less, a more tailored solution, would allow for scalability in the future.

 

I don't know much about SAN connectivity, from the little I've researched, a full Fibre connection would be best for us I think.

 

How equipment would we need to purchase as well as the actual SAN to get it up and running? Or is it simply a case of buying a SAN, buying and fitting a fibre connection to the nominated server and that's it for the hardware?

 

I've probably forgot some other questions I had, so I may come back with random questions. I have read some of the other relevant threads made about SANs in the past year, but with my lack of knowledge on the subject, I'm not really understanding it.

 

Any help would be greatly appreciated.

[SYNACK]

SANs can be tricky but are reasonably simple if you break the problem down.

 

The first as you have pointed out is connectivity, there are three main options and which you choose depends on your budget and setup. The fastest and most expensive is FibreChannel which requires a SAN switch and dedicated cards on each server. The next fastest is SAS which uses external SAS cables and pushes 3GB/s per chanel, this also required dedicated cards per server unless you have specific late model hp ones. This is much cheaper than fibrechannel though. Next up is iSCSI which really should have its own switch but uses the much cheaper ethernet standard. This is limited by ethernet pushing 1GB/s, some of the higher end SANs will allow you to aggregate but this will not scale as well as fibrechannel or SAS. On the cheaper side for speed and cost we ended up with a SAS one for a primary school here.

 

Next up is storage capacity, you can choose SANs that have only the bays provided or ones that support drive shelves. There is not need to use all the bays but the more drives you have the faster it is. For expandability you want one that supports extra drive shelves that you can add later if your needs increase.

 

Storage speed is determined internally by how many drives you have and what standard they support, SSD is the fastest but insane expencive. SAS dual port is the next fasted followed by SATA. Most new arrays you can actually mix standards so long as only one standard is used per volume.

 

Features like snapshotting and mirroring are also something to look into but that depends on your requirements.

 

SANs are offered by all sorts of providers, HP, SUN, EMC, even IBM (not recommended from my experience).

Edited January 28, 2011 by ChrisH

[Tricky_Dicky]

Have you spoken to Zentek about this? I've often found that if you have a good relationship with a supplier they can help and guide you. Even though they might not be the cheapest, having the support and backup for a large project can be worth a lot.

 

Dell also supply SANs and are offering some very impressive pricing at the moment.

 

Regards the switch, ideally you would want 2 so you have no single point of failure but it can depend on budget and also 1 high quality switch may well be a lot better. I believe as well that you can push 10gb with iSCSI over the correct switch.

 

Rich

[CHiLL]

SANs can be tricky but are reasonably simple if you break the problem down.

 

The first as you have pointed out is connectivity, there are three main options and which you choose depends on your budget and setup. The fastest and most expencive is FibreChannel which requires a SAN switch and dedicated cards on each server. The next fastest is SAS which uses external SAS cables and pushes 3GB/s per chanel, this also required dedicated cards per server unless you have specific late model hp ones. This is much cheaper than fibrechannel though. Next up is iSCSI which really should have its own switch but uses the much cheaper ethernet standard. This is limited by ethernet pushing 1GB/s, some of the higher end SANs will allow you to aggregate but this will not scale as well as fibrechannel or SAS. On the cheaper side for speed and cost we ended up with a SAS one for a primary school here.

 

Next up is storage capacity, you can choose SANs that have only the bays provided or ones that support drive shelves. There is not need to use all the bays but the more drives you have the faster it is. For expandability you want one that supports extra drive shelves that you can add later if your needs increase.

 

Storage speed is determined internally by how many drives you have and what standard they support, SSD is the fastest but insane expencive. SAS dual port is the next fasted followed by SATA. Most new arrays you can actually mix standards so long as only one standard is used per volume.

 

Features like snapshotting and mirroring are also something to look into but that depends on your requirements.

 

SANs are offered by all sorts of providers, HP, SUN, EMC, even IBM (not recommended from my experience).

We wont have problems with network load, as we have a 20Gb fiber links between the core switches, 10Gb fiber links between the core switches and the client switches, then 1Gb Ethernet links from the client switches to the workstations. With that in mind, plus my bosses recommendation, FC is the preferred method.

 

Howcome a fibre switch is needed between the server and the SAN box? Can't they be directly connected? What does the fibre switch do?

 

Since we're a school, obviously our demand is at specific times, logon and logoff at the hour are the main ones. The SAN would accomodate all data, from profiles, to home folders, to resources.

 

We have a VMWare ESX server which is running our terminal server. I've just read that this can manage SAN. Is this recommended?

 

Have you spoken to Zentek about this? I've often found that if you have a good relationship with a supplier they can help and guide you. Even though they might not be the cheapest, having the support and backup for a large project can be worth a lot.

 

Dell also supply SANs and are offering some very impressive pricing at the moment.

 

Regards the switch, ideally you would want 2 so you have no single point of failure but it can depend on budget and also 1 high quality switch may well be a lot better. I believe as well that you can push 10gb with iSCSI over the correct switch.

 

Rich

If we could afford such equipment, then we would, but the cuts made in this country aren't helping our cause. I will contact Zentek, but I'm just having a poke around in the subject.

Edited January 28, 2011 by CHiLL

[Duke]

You'll find a lot of love for the Oracle Sun S7000 range on these forums, definitely worth a look. There's a huge thread (I think it has 7110 in the title) that covers a lot of the general questions.

 

If you're replacing Windows file servers and you don't want to route your traffic through a Windows server (adds an extra bottleneck and point of failure) then technically what you're looking for is a NAS (file-level, NFS or CIFS/SMB) rather than a SAN (block-level, iSCSI). The nicer options out there will cover both of these as 'unified storage'.

 

I've been down the NetApp route an regretted it (£10,000 for an NFS licence? No thanks. Extra disk tray for the same price as an entire S7000? No thanks again). I have mixed feelings on EMC. HP and the stuff Dell resell are okay depending on your requirements. The IBM stuff is targeted at a pretty specific market. Personally I think S7000 is money well spent though.

 

We're a semi-reference-site for S7000, PM me if you want to chat and I'll give you my details or I can put you in touch with people at Oracle or resellers. I've got a lot of material/research from the past couple of years you might find useful too. (I don't get anything for this, I just think it's good kit).

 

Cheers,

Chris

[Tricky_Dicky]

Can I have your network please.....

[Duke]

Can I have your network please.....

 

Must admit I WTF'd at someone in a school already running multiple bits of 10Gb kit and 1Gb to every desktop...

[CHiLL]

You'll find a lot of love for the Oracle Sun S7000 range on these forums, definitely worth a look. There's a huge thread (I think it has 7110 in the title) that covers a lot of the general questions.

 

If you're replacing Windows file servers and you don't want to route your traffic through a Windows server (adds an extra bottleneck and point of failure) then technically what you're looking for is a NAS (file-level, NFS or CIFS/SMB) rather than a SAN (block-level, iSCSI). The nicer options out there will cover both of these as 'unified storage'.

 

I've been down the NetApp route an regretted it (£10,000 for an NFS licence? No thanks. Extra disk tray for the same price as an entire S7000? No thanks again). I have mixed feelings on EMC. HP and the stuff Dell resell are okay depending on your requirements. The IBM stuff is targeted at a pretty specific market. Personally I think S7000 is money well spent though.

 

We're a semi-reference-site for S7000, PM me if you want to chat and I'll give you my details or I can put you in touch with people at Oracle or resellers. I've got a lot of material/research from the past couple of years you might find useful too. (I don't get anything for this, I just think it's good kit).

 

Cheers,

Chris

Cheers, I'll have a look at those. Wish these companies would put the prices on the website, would make life easier!

 

Can a SAN not be connected to our network switches? (I know they're different technologies/protocols, but my knowledge isn't great on this as I've said)

 

This, and the network are for future uses. We could start doing video editing, which would require the network speeds and disc space to support that. Not now, but maybe in 5 years or something like that.

 

I must admit, the network outshines the workstations, some (only a few) will only support 100Mb unless we fit aftermarket network cards. Most are gigabit though.

 

For an entry level SAN, what prices are we looking at here? 10TB or something. HP are offering 7.2TB for £20,000.

 

Got a few other questions coming your way;

 

Some SANs come with a server built-in, so how would that effect its connectivity with the network?

What actually governs the discs inside the SAN?

I thought NAS was simply hard drives connected together with an Ethernet connection.

How does SAN differ from NAS?

In a school environment, which one would be most effective for the money?

Edited January 28, 2011 by CHiLL

[Duke]

Cheers, I'll have a look at those. Wish these companies would put the prices on the website, would make life easier!

 

It's nice kit but you may be better off looking elsewhere if you definitely want Fibre Channel. I believe the S7000 has FC support but there are probably dedicated solutions that might fit your needs better. No need to run FC if you've already got good Ethernet structure though, it's just an extra layer of hassle and you'll need new switches.

 

Can a SAN not be connected to our network switches? (I know they're different technologies/protocols, but my knowledge isn't great on this as I've said)

 

Yes, they definitely can. All the stuff I mentioned (SAN/NAS, CIFS/SMB/iSCSI) are IP-based network protocols and will work fine through standard switches (either fibre or copper). 10Gbe modules are available for the S7000 if you want to take advantage of your core switches capacity. Fibre Channel is something else entirely and will need specific adaptors and switches.

 

This, and the network are for future uses. We could start doing video editing, which would require the network speeds and disc space to support that. Not now, but maybe in 5 years or something like that.

 

I would really strongly make sure you get a SAN/NAS solution that is expandable then. This is where most of the cheaper options fall down because you can only put larger disks in them (wiping the storage in the process), the better solutions will take a large amount of disk trays, each with 24 (usually) disks. The good stuff should easily scale into the Petabyte range.

 

For an entry level SAN, what prices are we looking at here? 10TB or something. HP are offering 7.2TB for £20,000.

 

Depends entirely on the manufacturer. Ballpark figure for an entry-level S7000 that I have on hand is £18k list for 12TB, about £21k for 24TB including SSD accelerators. (those capacities are all 'raw' storage figures)

 

I paid £37.5k for our S7410 (24TB), NetApp quoted me £120k for a smaller, slower device.

 

Some SANs come with a server built-in, so how would that effect its connectivity with the network?

 

That's a bit of a vague description. Technically any SAN or NAS has a 'server' part to it (often referred to as a head), without that it's just a disk tray (DAS - Direct Attached Storage - usually SAS connectivity these days). The head will determine what the device can do and what protocols it can offer (iSCSI, NFS, SMB/CIFS, HTTP, FTP, etc.). The S7000 runs Solaris as the underlying OS (don't worry, it's all managed through an awesome web GUI - one of the best features of the product) so is very powerful.

 

What actually governs the discs inside the SAN?

 

The 'head' or controller, which usually controls the RAID and manages the disks, etc.

 

I thought NAS was simply hard drives connected together with an Ethernet connection.

 

Kind of. NAS runs at the file-level (see next question) and still requires a controller/head, and uses protocols like CIFS/SMB (i.e. Windows shares). A good NAS has AD integration so all the Windows file permissions and authentication are handled by the device.

 

How does SAN differ from NAS?

 

NAS = File-Level, which means the device using the NAS (e.g. a Windows PC) sends the files to the NAS and the NAS determines how they are stored.

 

SAN = Block-Level, which means the SAN presents raw storage to a device which then formats it to whatever filesystem it wants and it writes those files at the block-level. The SAN itself will still write those files to the disk, but in the way the remote device wants.

 

In a school environment, which one would be most effective for the money?

 

Depends on your exact environment.

 

These days, unified storage is fairly common and NetApp or EMC will sell you a device that can use CIFS/SMB, NFS or iSCSI as you need. The nice thing about the S7000 is all the protocols are included with no extra licensing.

 

EDIT: Disclaimer, before anyone says anything - I know some of my definitions aren't 100% accurate, I was generalising slightly and glossing over details to save typing.

 

I'll PM you in a sec.

 

Chris

Edited January 28, 2011 by Duke

[teejay]

Duke, you beat me to it!

Another recommendation for the Oracle stuff here, cracking kit and competively priced for what it is, especially when you consider there are no licencing costs for 'extras' or new features which you can get charged for on other ones.

I really would go for a unified storage device, the description in your OP looks like you really need for of the NAS type facilities than the SAN type facilities, so you would move all your file shares onto the storage device which serves these files directly without touching your windows file servers, leaving them to concentrate on being app servers. If you then want to virtualise your app servers, that's when you want the SAN facilities to provide shared storage for the virtual server images to enable failover and redundancy.

You 'could' use a SAN for the file server side, but this is really just like attaching an extra hard drive to your file servers as they will still be handling all the file serving traffic, if you want to remove that load from the servers you need the NAS route.

[Butuz]

IMHO forget fiber channel. You already have a 10gbit capable core switch (fair play I am jealous) so it would be much more cost effective and barely slower for you to use 10Gbit connectivity straight from the core into the san be it via Fibre or Cat6. Just put the storage server and data on a seperate VLAN. I assume you have 10Gbit connections to your servers too? Just tag on the SAN vlan using existing 10Gbit. Job done super fast.

 

Use a mixture of 1) SSD's for database data 2) 10/15k RPM SAS for general file storage and 3) 7200RPM SAS/SATA for largescale video files / backups etc.

 

Butuz

[teejay]

Oh, btw, if you are interested in the Oracle stuff, have a chat with HebdenLad on here, he works for Oracle Storage, top bloke (when he remembers to bring the biscuits, cough).

 

Edit: Also, speak to the guys from The Cutter Project on here, they are the resellers for the kit so can give you pricing info.

[CHiLL]

I do appreciate your help.

 

I've just done some research, and iSCSI seems to perform better, and be more cost effective, so it is looking like that's the best option. You mentioned that NAS determines how the files are stored...does that create a hit on performance? Our switches are more than capable of supported 10Gb, which is the preferred connection speed. As long as there's 10Gb Ethernet SANs or NASs, they're looking the bees knees at the moment. (Sorry for the pun!)

 

So if we bought a SAN or NAS, we could plug it into our switch at 10Gb, and manage it remotely?

If we bought a SAN, but used Ethernet, could that also be directly connected to a switch like a NAS?

 

IMHO forget fiber channel. You already have a 10gbit capable core switch (fair play I am jealous) so it would be much more cost effective and barely slower for you to use 10Gbit connectivity straight from the core into the san be it via Fibre or Cat6. Just put the storage server and data on a seperate VLAN. I assume you have 10Gbit connections to your servers too? Just tag on the SAN vlan using existing 10Gbit. Job done super fast.

 

Use a mixture of 1) SSD's for database data 2) 10/15k RPM SAS for general file storage and 3) 7200RPM SAS/SATA for largescale video files / backups etc.

 

Butuz

The connections from the switch to the servers are 1Gb each. The core switches are connected at 10Gb up and 10Gb down to take the load across the network. The reason for this is because our file servers are spread across two rooms on either side of the site.

Edited January 28, 2011 by CHiLL

[teejay]

Yes, most decent SANS have 10Gb options, no need for fibre channel unless you have a stack load of servers and then it's best to seperate the traffic and not use your main switch for this. The Intel Fibre Channel over Ethernet looks a promising option, but I don't think it's around yet.

Honestly though, from your initial post, look at a unified device that combines both NAS and SAN, but from your requirements you want NAS over SAN,

[Norphy]

IMHO forget fiber channel. You already have a 10gbit capable core switch (fair play I am jealous) so it would be much more cost effective and barely slower for you to use 10Gbit connectivity straight from the core into the san be it via Fibre or Cat6. Just put the storage server and data on a seperate VLAN. I assume you have 10Gbit connections to your servers too? Just tag on the SAN vlan using existing 10Gbit. Job done super fast.

 

Use a mixture of 1) SSD's for database data 2) 10/15k RPM SAS for general file storage and 3) 7200RPM SAS/SATA for largescale video files / backups etc.

 

Butuz

 

Don't put iSCSI on your core switch! Your iSCSI fabric should be completely seperate from the main network, whether the network is 10GBe or not.

[Duke]

I've just done some research, and iSCSI seems to perform better, and be more cost effective, so it is looking like that's the best option. You mentioned that NAS determines how the files are stored...does that create a hit on performance? Our switches are more than capable of supported 10Gb, which is the preferred connection speed. As long as there's 10Gb Ethernet SANs or NASs, they're looking the bees knees at the moment. (Sorry for the pun!)

 

Performance is dependant on what you're doing really. If you have Windows clients that need to talk directly to the SAN/NAS (e.g. for home directories, profiles and shared resources) then you really need to be running CIFS/SMB (i.e. Windows shares) direct from the NAS. The alternative way to do it would be to offer iSCSI to a Windows server then have that server offer the file shares. However at that point you've pretty much wiped out any performance benefit you might see from iSCSI by running it through the Windows TCP/IP stack overhead.

 

So if we bought a SAN or NAS, we could plug it into our switch at 10Gb, and manage it remotely?

 

Yep. The type of management depends on the device. NetApp stuff has a web interface but you also do some stuff through a Windows MMC snap-in and some stuff through the CLI. The S7000 has a really nice web interface, and you can run that on a dedicated 1Gb or even 100Mb port if you want and use other ports for actual data traffic.

 

If we bought a SAN, but used Ethernet, could that also be directly connected to a switch like a NAS?

 

Yes, the main SAN protocol (unless you're messing with FC) is iSCSI, which is SCSI data encapsulated in an IP packet, and runs fine alongside all other standard IP network traffic.

 

Chris

Edited January 28, 2011 by Duke

[Duke]

The Intel Fibre Channel over Ethernet looks a promising option, but I don't think it's around yet.

 

I remember looking at FCoE when it was first talked about years ago and it looked great, but nothing seems to have come of it. Now 10Gb Eth is starting to become common I don't think there's going to be any huge advantage to FCoE for most of us who are using NFS/CIFS/iSCSI.

[CHiLL]

Don't put iSCSI on your core switch! Your iSCSI fabric should be completely seperate from the main network, whether the network is 10GBe or not.

This could be a problem, as our two server rooms both house core switches. I thought it was an Ethernet connection from NAS box to a 10Gb connection on the core switch. I would be on a different module, but still in the same switch. What's the reasoning for this? Also, it is only our core switches that support 10Gb.

 

Performance is dependant on what you're doing really. If you have Windows clients that need to talk directly to the SAN/NAS (e.g. for home directories, profiles and shared resources) then you really need to be running CIFS/SMB (i.e. Windows shares) direct from the NAS. The alternative way to do it would be to offer iSCSI to a Windows server then have that server offer the file shares. However at that point you've pretty much wiped out any performance benefit you might see from iSCSI by running it through the Windows TCP/IP stack overhead.

We would be holding home folders, profiles and shares on it.

 

Yep. The type of management depends on the device. NetApp stuff has a web interface but you also do some stuff through a Windows MMC snap-in and some stuff through the CLI. The S7000 has a really nice web interface, and you can run that on a dedicated 1Gb or even 100Mb port if you want and use other ports for actually data traffic.

Sounds good!

 

Yes, the main SAN protocol (unless you're messing with FC) is iSCSI, which is SCSI data encapsulated in an IP packet, and runs fine alongside all other standard IP network traffic.

 

Chris

Sounds even better!

Edited January 28, 2011 by CHiLL

[Duke]

Don't put iSCSI on your core switch! Your iSCSI fabric should be completely seperate from the main network, whether the network is 10GBe or not.

 

I don't see the problem with this as long as it's on a different VLAN/subnet/IP range? Don't get me wrong, if your SAN/NAS was generating Gbps of traffic and your core switch was under-specced then you'd have issues, but if you have the bandwidth and you segregate the traffic appropriately then I can't see the issue.

 

It's very likely your NAS traffic will need to talk to devices on the core switch (other servers) so I don't see how you'd avoid it. If you're using the SAN/NAS for virtualisation (i.e. virtual machine storage), or perhaps to host OLTP-style databases then yes I would definitely put it on another VLAN away from my normal network traffic, but it could still be on the same physical switch?

 

In an ideal world you'd be able to keep different traffic on different physical switches - and for connecting a SAN to VMware/Xen/Hyper-V hosts it would certainly be possible - but other than that I think iSCSI via the core switch should be reasonable?

Edited January 28, 2011 by Duke

[Duke]

We would be holding home folders, profiles and shares on it.

 

Go NAS straight to your clients via CIFS - anything else is losing out on so many of the benefits of a SAN/NAS, and if it's done properly your clients will have no idea they're not talking to a Windows server.

[CHiLL]

I don't see the problem with this as long as it's on a different VLAN/subnet/IP range? Don't get me wrong, if your SAN/NAS was generating Gbps of traffic and your core switch was under-specced then you'd have issues, but if you have the bandwidth and you segregate the traffic appropriately then I can't see the issue.

 

It's very likely your NAS traffic will need to talk to devices on the core switch (other servers) so I don't see how you'd avoid it. If you're using the SAN/NAS for virtualisation (i.e. virtual machine storage), or perhaps to host OLTP-style databases then yes I would definitely put it on another VLAN away from my normal network traffic, but it could still be on the same physical switch?

It would certainly be a VLAN of its own.

 

Go NAS straight to your clients via CIFS - anything else is losing out on so many of the benefits of a SAN/NAS, and if it's done properly your clients will have no idea they're not talking to a Windows server.

I'll have to research CIFS and SMB. I've heard of them, but I can't remember what they are. As for the server part, that sounds interesting. If AD is imported correctly, and once it is configured, it sounds like it will run itself!

[Duke]

I'll have to research CIFS and SMB. I've heard of them, but I can't remember what they are. As for the server part, that sounds interesting. If AD is imported correctly, and once it is configured, it sounds like it will run itself!

 

CIFS and SMB are just the technically correct names for what Windows shares use. Presumably at the moment your users access data via something like \\server\sharename or even DFS paths like \\domain.local\DFSroot\sharename? Even if they actually get to that path via a mapped drive? That's all CIFS/SMB (same thing) are, but most SAN/NAS companies use this technical term. I can do \\SAN\sharename with no problems. Ever used Samba on a Linux machine to access a Windows server? Same thing.

 

It's not even a AD import - the SAN/NAS will communicate directly with one of your domain controllers and run queries in real-time - the same way a Windows server would. I can set permissions on a file or folder that's hosted on the NAS just the same way I would on a file or folder that's hosted on a Windows server, and when someone on the domain tries to access the file/folder the NAS will query their AD groups and work out the permissions.

[teejay]

It would certainly be a VLAN of its own.

 

 

I'll have to research CIFS and SMB. I've heard of them, but I can't remember what they are. As for the server part, that sounds interesting. If AD is imported correctly, and once it is configured, it sounds like it will run itself!

 

Yep, it does and is far quicker than Windows file servers. We run all our file shares from a Oracle S7310 and that replaced at least 7 Windows file servers, doesn't even break into a sweat and that's just running via 3 x 1Gb links using link aggregation, stick a 10Gb nic in and it will fly :-)

BTW, what make are your switches?

Also, if you fancy a trip up the motorway to West Yorkshire, you're more than welcome to come and have a look at what you can do. We use both the NAS and SAN facilities of our Oracle boxes, so can demonstrate both to you.

[CHiLL]

Yep, it does and is far quicker than Windows file servers. We run all our file shares from a Oracle S7310 and that replaced at least 7 Windows file servers, doesn't even break into a sweat and that's just running via 3 x 1Gb links using link aggregation, stick a 10Gb nic in and it will fly :-)

BTW, what make are your switches?

Also, if you fancy a trip up the motorway to West Yorkshire, you're more than welcome to come and have a look at what you can do. We use both the NAS and SAN facilities of our Oracle boxes, so can demonstrate both to you.

Our switches are HP Procurves. Can't remember the models.

[Soulfish]

Thought I'd chip into this thread. First things first there are only two forms of connectivity for SANs - Fibre Channel or Ethernet. SAS is only used for DAS setups (Direct Attached Storage). If you don't have an existing Fibre Channel setup (which you don't since your asking about it) it's easier and cheaper to go for iSCSI/Ethernet based connectivity. If you've made do without fibre channel until now, you won't miss it and you'll save the cost of Fibre Channel HBAs, and Fibre Channel switches/directors. As others have discussed there is the possibility of utilising a converged network infrastructure (Fibre Channel Over Ethernet or FCoE), but unless your running something like the nice fancy Cisco Nexus range of switches I doubt your kit is capable .

 

So iSCSI/Ethernet SANs. This tends to be where a large segment of the SAN market is heading. Having tendered for a new (large) SAN this summer, and running two different SANs here I've had the pleasure of looking at virtually every manufacturer on the market (IBM, Hitachi, Sun/Oracle, EMC, NetApp, HP EVAs, Dell, Dell Equalogic, Lefthand (HP P4000), Pillar, Huawei/Symantec, 3Par, Compellent and lots of others). As mentioned previously a unified storage system combining NAS and SAN functionality gives you the best of both worlds. You get to serve files directly from the device, while still getting to do block level access as well. There are lots of other nice things you can look for as well - automated tiering between storage tiers (moving commonly access files to fast disks, least access files to slow disk), snapshotting (allowing for quick access to previous versions), replication and finally the use of SSDs/Cache to speed everything up.

 

Like lots of other people I'm a big fan of the Oracle S7000 series of SANs. They're reasonably priced with a very easy to use GUI and good performance. The best thing you can do in this market place however is play the manufacturers off against each other. Unless your getting 60+% off list you haven't got a good enough deal . Dell have just bought Compellent, and EMC have just launched a new unified product targeted at SMBs so there are some very good deals available.

 

We currently run a Sun/Oracle 7110 and an EMC Celerra NS-480. Both are connected to a physically separate storage network that consists of 2 x Juniper EX2500 24-port 10GbE switches. Each server then has an Intel X520-DA2 10GbE card and we use direct-attached twinax SFP+ cables from the servers to the switches. The Sun 7110 has around 2tb raw while the EMC has around 120tb raw. Unless your pushing lots and lots of data then a good gigabit switch would be fine for most SANs. A separate switch is preferable, but a VLAN on your core switch can also work.

 

Finally I know you mentioned at the start about buying the SAN/disk shelves part empty and filling them up later. If you can buy any disk shelves full. They'll be a lot lot cheaper to buy at the start then to fill later, and unfortunately you can't just buy the disks and put them in yourself. I don't know of any vendor that allows you to buy the disk caddies and some of them use fibre channel disks which makes it even harder!

 

If you've got any questions feel free to ask or drop me a PM