Clustered SAN

[hotwired007]

I have 2x HP Procurve ML115 Servers (Opteron 1352 (Quad 2.10Ghz 4x 512mb) 4GB DDR2 RAM with 4x WD RE3 1TB HDD's and a 16GB Lexar Jumpdrive.

 

I'm trying to set these up to run as a SAN with Load balancing and fail-over, using the USB as the boot drive and 3 of the HDD's in a RAID 5 array or an XFS+1 Array and the last drive as a hot swap.

 

I was planning on using NexcentaStor 2.0.0 Develeper Edition but then realised that i could only set each server up as a seperate SAN and not cluster them... I'm now at a bit of a sticky spot so any advice or ideas would be great!

 

I'm looking at Openfiler atm, but having dificulty loading it onto the USB drive. Also been looking at FreeNAS but without any joy on how to cluster them...

 

Thanks in advance!

[hotwired007]

After some research and test installs freeNAS looks the most promising, i have both servers installed with FreeNAS and each have a share accessable over the network.

 

SAN01/DATA & SAN02/DATA.

 

I'm now trying to setup UNISION to syncronise the shared data folders across the two FreeNAS servers. Does anyone know how to do this?

[CyberNerd]

 

I'm now trying to setup UNISION to syncronise the shared data folders across the two FreeNAS servers. Does anyone know how to do this?

 

you shouldn't need to synchronise the data.

If you set up the file system using a clustered filesystem such as GFS then they should be able to read and write to the same SAN disk simultaneously.

clustering without GFS isn't really clustering

[hotwired007]

The filesystem is UFS (GPT and Soft Updates) the default for FreeNAS.

 

All the information points to using UNISON or RSync to syncronise the two shares.

[CyberNerd]

ok, thats not really clustering - you won't be able to use the two servers simultaneously if you just copy the data periodically.

 

I would go with rsync - it's pretty easy to set up and you can tunnel it over ssh.

rsync

 

redhat.com |

if you were to do it again...

[hotwired007]

my understanding with unison was that it was a constant syncronisation.

 

I intended on using the servers to store a number of VMs on each and for example have 8 VMs total but run 4 each from the individual NAS boxes...

 

Unfortunately its a difficult situation although i'm now looking at OpenFiler to run a HA Clustered environment. I'm goign to buy a couple more HDDs for the Servers and trial them with it although they wont turn up til tommorow so i'll keep trying with the FreeNAS system.

[Guest Guest]

Unfortunately its a difficult situation although i'm now looking at OpenFiler to run a HA Clustered environment.

 

Again it isnt really clustering in how you are thinking of it, its just a backup server which is a straight mirror of the primary using DRBD. When the primary goes down the HA part kicks in and promotes the backup, which takes over the services. Works really well, but only 1 server can be a target at a time in DRBDs normal operation. (IIRC you can couple it with a cluster FS and have 2 primaries, but that certainly isnt its default operation)

 

In other words; HA = yes. Backup = yes, Load balencing = No.

For load balencing you need a cluster FS.

Edited July 29, 2009 by Guest

[CyberNerd]

straight mirror of the primary using DRBD

 

I recall dhicks built a HA virtual host using DRBD

DRBD:What is DRBD

 

That is realtime, as is GFS. I'm pretty sure unison is not (but I not used it)

[lucasmocellin]

Hello guys!

 

I'm doing the same for a FreeNAS storage. Looking for some redundancy - not necessarily fail-over.

 

I will build an environment with some VMware ESX machines and I need a reliable FreeNAS with redundancy - trying to eliminate SPOF (Single Point of Failure).

 

I already have a solution for the VMware, but not for the NAS.

 

I thought about DRDB and an exclusive 1GB link between them, but I'm not sure if it will have the appropriated performance. Maybe kinda of DRDB asynchronous between master and slave?

 

Can you both tell me about your environments? How is the final set up? How is the performance? What you think is the bottleneck? Did you guys do some reliability/performance analyzing on that?

 

Thanks in advance!

 

Lucas.

[lucasmocellin]

ah, I need to finish the planning of this project by wednesday. =(

 

sorry for the hurry, but if you guys can help me out with some opinions, I would really appreciate.

 

thanks again,

 

Lucas

[Guest Guest]

Are you going to be using SSDs? If not I doubt very much that you will cause a bottleneck in the 1gb link between the SANs/NASs. It will most likely be at your HDs. I was in the same boat too when I was first setting up DRBD so I know what you are thinking. dhicks has been running DRBD now for some years, drop him a PM.

 

There's absolutely tonnes of info on here about DRBD and aggregation of NICs, have a search.

 

I havn't looked at FreeNAS for quite some time but when I was setting up a few years ago Openfiler seemed far better suited for the job as it was targeted at enterprise where as freenas was for home users.

[lucasmocellin]

No SSD's. SATA and SAS HD's.

 

I need to use FreeBSD, and DRDB is not available for BSD, there is the equivalent, but not that reliable as I've read around.

 

FreeNAS looks very reliable now, and no much difference between them as I google (I've never worked with them, my opinion is just about googling).

 

I will send him a message! I would like to find someone who had set up a FreeNAS + any kinda system redundancy.

 

thanks!

 

lucas.

[dhicks]

Hello Lucas - I got your PM, had a look at this thread. As pointed out above, I've been running DRBD on assorted hardware for a while - at my previous school I had all our Xen VMs running on DRBD-mirrored volumes. This means you have local performance for disk read operations, only disk write operations have a delay (probably similar/identical as to what you'd get using a dedicated SAN server) as the written data has to go accross the network to the mirrored device. At the kind of scale that most schools operate at (a couple of servers hosting virtual machines), I think mirrored volumes make more sense than a central iSCISI server - you do have to buy (and power) double the number of harddisks, but then you save on a whole dedicated server and you get proper, full redundancy

 

DRBD's protocol works differently than iSCISI. As I understand it, iSCISI can make better use of aggregated network links - a single DRBD connection will not use more bandwidth than is available to one network port, i.e. a DRBD connection is limited to 1Gb/s even if running over an agregated 4Gb/s link. That said, as pointed out above, it is unlikly that will actually make too much difference to you - remember, only write operations need to be sent over the network anyway, so the bandwidth required by DRBD isn't really all that great.

 

As I've just recently discovered, DRBD's default setup (sensibly) assumes it is running on a general network, shared with other devices, and only uses up to 30% of the available bandwidth. If you have a dedicated storage area network (i.e. the original meaning of "SAN", although most people now seem to take that as meaninga large, central iSCISI server) you should set DRBD to use all the bandwidth it wants.

 

FreeNAS looks very reliable now, and no much difference between them as I google

 

I think OpenFiler actually has DRBD built-in for setting up mirrored devices.

 

How much / what sort of redundancy are you aiming for here? Bear in mind that mirrored block devices are not backup - delete a file on the master volume and DRBD will quickly and efficiently delete it on the secondary volume with a minimum of delay. Your best bet might be to use rsync to take regular (hourly?) backups of your primary file server, placing the backups on a secondary server and using deduplication to store them (i.e. for identical files just store a hard link to the original file, don't copy the original). Then, just make that server a Samba server with the same file permissions as the primary server, able to serve files directly to users. That way, you not only get a backup of your files that people can go to for previous file versions, you can also use that file server as the main file server in an emergency, maybe loosing an hour's work in the change-over. I underatand FreeNAS supports acting as an rsync server, so this should be easy enough to set up.

[maniac]

Not set one up, but I have done a fair bit of reading. This guide seems about the best I've found using open filer as it's prefered package - it's what I intend to follow when I try a simelar project in a few months time - just doing some proofs of concept on some old hardware to start with.

 

Setting up a High Availability NAS/SAN using Openfiler | Easy to Read Articles

[lucasmocellin]

Hi,

 

I also think mirrored disks are better for this situation, I'm looking for redundancy, not a backup. For backup I will use Bacula.

 

I just got that my hardware are not that good as I imagined, and they are not that proper also, as they support only up to 4 HD's.

 

so by now I'm almost giving up on the NAS/SAN and have a local RAID with a good backup.

 

my main problem with the NAS is that I would store all the system in there, so if I got a problem with the NAS, ALL THE SYSTEMS WILL SHUT DOWN! and this makes me think a lot about this decision.

 

as I didn't find a professional solution case, I will give up temporarily on that.

 

thanks a ton for your reply!

 

Lucas.

[Guest Guest]

I also think mirrored disks are better for this situation

 

 

You looked at using RAID 10? I'm just purchasing some servers now to be virtual hosts with local storage and that's the way I'm going.

[Richie1972]

i wanted to go down the clustered SAN route, but didn't have the money.

If I did have the money I'd have gone with two storage servers with Windows Storage Server on or Server 2008 R2 with Starwind Server iSCSI targets that do support clustered environments.

Link the servers with 10gb (or even bonded 10gb) LAN connections (possibly even 10gb fibre to host one SAN in a different building to meet disaster recovery requirements - along with possibly one or more 2008 R2 cluster servers hosting Hyper-V)

[PeteHopton]

OK, there is an easy way to achieve this, but it's not that cheap.

 

Starwind HA Enterprise Edition will do this very fast and very reliably from a Windows based OS, I've been using it for years and can vouch for it.

 

iSCSI SAN Software and Storage Virtualization Solutions

 

They might do you an education discount.