Homespace Server Using CIFS/NFSv4/ZFS

[haute]

Hello!

 

Hopefully, there are people around these parts who can help out.

 

I am an admin in the Comp Sci dept at a medium-sized university, and we have just recently started a storage infrastructure upgrade.

 

We have been using a NetApp for the last few years, and we have finally had enough of it tanking all time along with NetApp's thoroughly asinine stance where they "do not support CIFS from Linux". Since we have an uneven split of our desktops (20% Windows, 80% Linux), we need to support CIFS (from Windows and Linux) and NFSv4 from Linux in order to provision home spaces for our faculty and grad students.

 

We purchased a Dell EqualLogic last year for our VMWare infrastructure, and are now in the position of having to build a server which can be a ubiquitous file server using the EqualLogic for its file system. Since this needs to be an extensible setup, we came to the conclusion of needing/wanting to use ZFS. That way, should we need more space to use via another EqualLogic, we can just add it to our zpool and be on our merry way. The other caveat is that snapshots under Linux with the EqualLogic are a no-go, so ZFS ftw again in that respect.

 

The trouble we are having is we dont know how to go about setting up a heterogenous home space server with snapshots where clients access their data from both CIFS/NFS seamlessly and are auth'd against Active Directory on the Windows side, and LDAP on the Linux side, and all UID's/GID's are correctly mapped (all usernames are exactly the same on each side, btw).

 

Here is what we have looked into so far:

 

1) OpenIndiana - no paid support, and a fairly convoluted setup. Not only that, any version of Solaris has a very narrow implementation of LDAP authentication, so we'd have to somehow shoe-horn it to communicate with our campus LDAP. Aside from that, we cannot get paid support at any point should we need it, or be able to afford it.

 

2) Nexenta - absolutely no LDAP support which will work with our campus LDAP, and no build environment installed on it, so we couldnt even build pam_ldap if we wanted to. Couple that with the fact Nexenta's virtual NICs do no work with iSCSI in VMWare ESXi (where we are creating the server), this is no longer an option. We were actually told by them that if we implemented an LDAP integration which wasnt theirs, we would not be supported b/c our system would be non-standard. Fail.

 

3) OpenFiler - no ZFS. No NFSv4. Fail.

 

4) Windows 2008 R2 w/ Services for UNIX - requires an NIS server for UID/GID mappings from Linux to Windows. Our campus NIS server holds no authentication credentials, so that kills this project. NTFS over ZFS? No way.

 

5) FreeNAS - LDAP? Check. AD? Check. ZFS? Check. NFSv4? Check. Do LDAP and AD actually work? No. *sigh*

 

6) Solaris - $$$ which we cannot afford.

 

7) OpenSolaris - cannot be used for production purposes, LDAP...

 

The trouble is right now, we have zero budget, and the possibility of paying for something which is a solution is non-existent. Our faculty is facing a massive budgetary shortfall (in the millions) due to funding cuts, so we have to find something which is literally a drop-in replacement for our NetApp.

 

If anyone can point us to something which can get us off the ground, we'd be *very* appreciative, b/c we've spent literally months banging on each of the list above here, and each one has a major fault.

 

TIA!

Edited April 14, 2011 by haute

[joegalliano]

Hi,

 

I know Dell are working on clustered nas with Patents aquired from Exastor & they are working to integrate this with their Eqlogic & Compellent SANS. I looked at Exastor a few years ago and its has ISCSI, FC, NAS i,e Cifs & NFS. Dells Compellent SAN also has ZFS and Cifs NAS available now, not sure if you can run the two together though.

 

May also be worth keeping your netapp for now and possibly look at something like Falconstor or another storage virtualization product until Dell have got their products sorted.

 

1 last option may be Open E, but I am not sure if it has support for ZFS if that is required.

 

Data Storage Software V6 (DSS V6) - Comparison | Open-E

[joegalliano]

Just checked Compellen't ZNAS runs ZFS, CIFS & NFS

[Soulfish]

Compellents ZNAS is just Nexenta rebadged connecting back through to the normal compellent infrastructure . I don't know if it'll do what you want (it's based on OpenSolaris) but have you looked at the Oracle S7000 range? Used to be Sun open storage. A few people here (including myself) have used them and they're pretty decent boxes (on the whole!).

[haute]

Thanks for the replies.

 

 

Unfortunately, all of them are paid for options, and we need to implement something which is OSS/GPL/CDDL.

 

 

The VSS option appears to only to NIS/AD UID/GID mapping in the paid for version, which doesnt help anyway as we dont have NIS....

 

If we could find a detailed how-to using OpenIndiana, that might be acceptable, but since OpenSolaris has all but tanked, documentation for such a thing is almost non-existent.

[joegalliano]

Have you considered Ubuntu running ZFS, Samaba and authentication serviceswith DAS storage perhaps to keep down overheads

[haute]

Have you considered Ubuntu running ZFS, Samaba and authentication serviceswith DAS storage perhaps to keep down overheads

 

We thought about it briefly, but ZFS on Linux is actually run via FUSE and the performance is *cough* less-than-stellar.

 

Further to that, we have had a lot of trouble binding Linux clients to our AD and mapping the UIDs/GIDs for any UNIX resource to those of the Windows equivalent.

[dhicks]

Further to that, we have had a lot of trouble binding Linux clients to our AD and mapping the UIDs/GIDs for any UNIX resource to those of the Windows equivalent.

 

Here's a couple of threads in which I stumble towards a workable solution in getting a Samba-based file server up and running:

 

http://www.edugeek.net/forums/nix/49844-zfs-deduplication-dedup-vs-compression.html

http://www.edugeek.net/forums/nix/48032-configuring-samba.html

 

I think, in the end, I had Samba working with UIDs/GIDs properly - you just have to be consistent accross file servers as to how you're going to map them. I gave up on ZFS in the end, I found it more trouble than it was worth. I set our servers up using LVM - would that not be acceptable for your extensibility and snapshot requirements?

 

Would BTRFS be suitible instead of ZFS, as supported by the latest release of Debian?:

 

Btrfs - Debian Wiki

 

Do you actually need to be able to pool the storage of future servers? Each of your users has a mapped home directory, either in Unix, Windows or both, on a particular server. Can't you just have some mappings pointing at one server and others at another?

[haute]

I think, in the end, I had Samba working with UIDs/GIDs properly - you just have to be consistent accross file servers as to how you're going to map them. I gave up on ZFS in the end, I found it more trouble than it was worth. I set our servers up using LVM - would that not be acceptable for your extensibility and snapshot requirements?

 

The trouble is that not all servers involved in the chain of authentication are under our control, so UID/GID mapping can get a little convoluted.

 

I suppose we could use LVM, but honestly, I havent looked into it as I have tried to steer us towards ZFS.

 

Would BTRFS be suitible instead of ZFS, as supported by the latest release of Debian?:

 

Btrfs - Debian Wiki

 

Never heard of it until now. I will take a look!

 

Do you actually need to be able to pool the storage of future servers? Each of your users has a mapped home directory, either in Unix, Windows or both, on a particular server. Can't you just have some mappings pointing at one server and others at another?

 

Yes, we need the capability of making the file system for user homes transparently extensible. That was the main reason for choosing ZFS; zpool add and you're off to the races.

 

So, if we ever ran out of space on our EqualLogic, we can just buy another one, provision another iSCSI device off of it and add it to the zpool on the server.

[haute]

I took a look at btrfs.

 

Unfortunately, it looks to be far too immature to implement in a production research environment.

 

It doesnt even have a working implementation of fsck at this point...

 

 

Otoh, it does look promising for the future, just not at this point in time.

[bossman]

What is the problem with LDAP and AD in Freenas?

 

As you have stated Freenas ticks all the boxes and then you state it fails why?

 

We have used it here for quite some time only for simple storage though would like to know why you cannot use in your environment.

 

Sorry I have no answers to your question but just being inquisitive as we are looking to integrate it with our Sun 7110 as a VM backup service and HA storage.

Edited April 15, 2011 by bossman

[Arthur]

Would FreeBSD v8.2 be suitable?

[mavhc]

freebsd will end up being the default zfs platform, plus it should run everything you need, no idea of the details though

[dhicks]

I suppose we could use LVM, but honestly, I havent looked into it as I have tried to steer us towards ZFS.

 

But what are the specific features of ZFS that you need/want that no other system can provide? Deduplication?

 

So, if we ever ran out of space on our EqualLogic, we can just buy another one, provision another iSCSI device off of it and add it to the zpool on the server.

 

You should be able to do the exact same thing with LVM - have a central server (or virtual machine) that connects to a number of iSCISI targets and pools them all into one large storage volume via LVM. I haven't tried this, but it'd be interesting to see what kind of overheads were involved, i.e. how much data winds up to-ing and fro-ing between the central controller and the iSCISI targets. I don't know if ZFS would fair better (although you'd kind of assume it would), but having a user's home space mapped to a specific server should have no such overheads at all. All you should need is a consistent hash to map a user's name to a given server and an associated script to ensure a new user's file area is created on the appropriate server and their home folder pointed to the right location. You'd need another script that re-arranged everyone's file storage when you wanted to add another chunk of storage to the scheme, but there shouldn't need to be too much downtime - use rsync to do an initial copy with the old user area still live, then take the old user area down and do another quick rsync to bring the new area up to date. If you ran this as an overnight job no-one should even notice.

[joegalliano]

Another free storage server which has ZFS, NFS, CIFs etc There is also a AD guide

 

http://sites.google.com/site/eonstorage/

[CyberNerd]

also look at GFS2 Chapter*1.*GFS2 Overview

[Arthur]

I was going to suggest EON too, until I looked through their FAQ.

 

What is the licensing on EON?

EON is free to use, copy, redistribute or modify for personal use. Business or any other use, must acquire the proper written consent or contract agreement. If you are a business, consultant, reseller or integrator please acquire the proper license agreement to use EON (eonstore @ gmail dot com).

Edited April 18, 2011 by Arthur

[haute]

Sorry for the radio silence, everyone.

 

I have been out of town on a small vacation since Thrs.

 

I will reply to what I can.

[haute]

What is the problem with LDAP and AD in Freenas?

 

As you have stated Freenas ticks all the boxes and then you state it fails why?

 

We have used it here for quite some time only for simple storage though would like to know why you cannot use in your environment.

 

Sorry I have no answers to your question but just being inquisitive as we are looking to integrate it with our Sun 7110 as a VM backup service and HA storage.

 

 

At this point, FreeNAS is not talking to our LDAP, nor our AD correctly.

 

On the LDAP side of things, we have a slightly non-standard LDAP setup, which can cause issues with clients that don't use the pam_ldap auth module from PADL software.

 

On the AD side, FreeNAS sometimes needs to be "massaged" to get it to hook up to AD. I don't have a link in front of me, but there are a few known issues with FreeNAS' AD binding setup. In order for us to get it to work, we'd have to do some hackery.

 

 

Would FreeBSD v8.2 be suitable?

 

Possibly, but we haven't fully investigated FreeBSD. The closest we have come is FreeNAS, and given FreeNAS' AD issues, I can assume those issues exist in the parent OS as well.

 

 

But what are the specific features of ZFS that you need/want that no other system can provide? Deduplication?

 

 

 

You should be able to do the exact same thing with LVM - have a central server (or virtual machine) that connects to a number of iSCISI targets and pools them all into one large storage volume via LVM. I haven't tried this, but it'd be interesting to see what kind of overheads were involved, i.e. how much data winds up to-ing and fro-ing between the central controller and the iSCISI targets. I don't know if ZFS would fair better (although you'd kind of assume it would), but having a user's home space mapped to a specific server should have no such overheads at all. All you should need is a consistent hash to map a user's name to a given server and an associated script to ensure a new user's file area is created on the appropriate server and their home folder pointed to the right location. You'd need another script that re-arranged everyone's file storage when you wanted to add another chunk of storage to the scheme, but there shouldn't need to be too much downtime - use rsync to do an initial copy with the old user area still live, then take the old user area down and do another quick rsync to bring the new area up to date. If you ran this as an overnight job no-one should even notice.

 

We are after zpooling, zraid, dedup. LVM just seems so cantankerous in my experience.

 

Plus, I don't know what the performance is like compared to ZFS. Another department went Solaris/ZFS/EqualLogic and are very, very happy, but we do not have the money to pay for a Solaris support contract from Oracle.

 

 

I was going to suggest EON too, until I looked through their FAQ.

 

Ya, saw that myself.

 

The other thing is it is based on OpenSolaris which means the LDAP implementation does not fit our environment.

[haute]

Originally Posted by dhicks

But what are the specific features of ZFS that you need/want that no other system can provide? Deduplication?

 

I forgot to add the most important feature: Snap shots.

[Guest Guest]

We are after zpooling, zraid, dedup.

 

Dedup is only available on ZFS since version 21. Last time I checked only Solaris/Opensolaris had this version; EON, Nexenta, Nexentastor etc

[haute]

Dedup is only available on ZFS since version 21. Last time I checked only Solaris/Opensolaris had this version; EON, Nexenta, Nexentastor etc

 

 

True. FreeBSD v8 is only at v13 for ZFS.

[Arthur]

The latest stable version of FreeBSD (v8.2) has ZFS 15 and there is a patch available which adds support for ZFS 28 from FreeBSD 9.

[mavhc]

You can't add more drives to a zpool if it's raidz