Sun 7410 Unified Storage + SAN/Userspaces Question

[Duke]

Hi folks,

 

First post, but I've been keeping an eye on the forums as a couple of people I've been working with are on here. I've recently bought a Sun 7410 Unified Storage System with a 22TB J4400 Array and three flash accelerators. I know there has been quite a bit of discussion on the Sun 7000 series, but it's been mainly aimed at the lower end 7110 and 7210 models, so if anyone has any questions about the 7410 then I'll do my best to answer them, although be aware I haven't fully deployed it yet in production so I won't have all the answers!

 

(Thanks to Andy at Cutter Project and Phil at Sun for their help, very good people to work with!)

 

On to my related question...

 

For those of you who have a SAN and Windows users with roaming userspaces/profiles, how have you got them saving directly to the SAN? At present we have three Windows servers with shares for userspaces and profiles, these locations are set in the user account in Active Directory and the 'My Documents' folder is redirected with Group Policy.

 

I'd like to have these users (1600+ of them) writing directly to the SAN to remove the Windows server overhead and point of failure. However, I'm starting to realise some issues like the ability to batch-create user shares and set permissions on them (easily done with our Windows user account tool), granular control of file and folder permissions, and quota management at the level provided my Microsoft's File Server Resource Manager.

 

If you're using a SAN with lots of Windows users, how do you create and manage the shares required on a large scale? I realise this isn't a problem with the Sun box (it fully supports AD and CIFS, and does no less than NetApp or EMC kit to my knowledge), but more a 'what's the best practice' question.

 

Many thanks,

Chris

[Soulfish]

Hi Chris,

 

I'm afraid I can't answer any of your questions (we've recently got our 7110 and haven't had a chance to play with it yet!), but I did just want to say that I'm insanely jealous of your 7410 and all that storage

 

How are you finding general performance out of interest?

[Duke]

Hi Soulfish,

 

The 22TB was a bit overkill, but when you consider that drops to 14TB in RAID6 and a format with two hotswaps, then the room you need for snapshots, it becomes a bit more reasonable! While we've got nowhere near 14TB of data on the network at the moment, I was planning for future expansion and using the SAN to store virtual machines for our virtualisation project.

 

I'd previously looked at NetApp, EMC, Dell, SANMelody and others for storage solutions, but at the promo pricing the Sun box was very hard to argue with. I already have a 4TB NetApp box so was seriously considering going with NetApp, but the Sun 7410 was basically half the price of a comparable NetApp 2050, plus all the licensing and OS is free!

 

I've only done some basic performance testing so far, and at the moment it's almost identical to a Windows box which simply tells me the bottleneck is somewhere else on the network. I'm looking into sorting out a better testing system as DTrace certainly hasn't shown the Sun box reaching any limits!

 

For those who might be interested and can offer any input, our long-term plan is:

 

• Pair of mirrored Sun 7410's
  
• Four Dell servers (Dual Nehalem Xeons + 36GB RAM) running VMware ESX (not yet tested, although Xen is very nice) in one location
  
• Six or eight older Dell servers running VMware ESX elsewhere on site for failover
  
• Virtualise the majority of our current 22 servers
  

 

Cheers,

Chris

[Ric_]

@Duke: I think Andy had asked me to tell you how I did this... and I then forgot

 

I found the easiest way was to create shares for the relevant things and asdsign the root full permissions for one user (me). I then created a folder in that share and dropped everything under that, using Windows to set the ACLs on those folders - I already had a script.

 

I used ROBOCOPY to copy the data off the old shares to the new ones and used the /SEC switch to keep the ACLs intact.

[Butuz]

Sounds like a good plan Chris. Might be worth testing Xen - I have it on good authority that it outperforms ESX currently.

 

Working on an order for a much scaled down version of exactly what you are planning.

Pair of mirrored Sun 7110's

Two Sun Servers (Dual 3.16 Quad Core Xeons + 16GB RAM each) running ESX / XEN

Two older RM Servers (Dual 2.33 Quad Core Xeons + 8GB RAM each) for redundancy.

 

Let us know how you get on!

 

Butuz

[Duke]

Ric - Thanks, that sounds good to me! I know that will definitely handle things like our Staff and Subject Resources folders and then we can just change the drive mapping for users.

 

'My Documents' userspaces are a little more tricky, at present it looks like this:

 

Share on server as username$

Username has full permissions

Staff have read-only permissions

FSRM sets a hard-limit quota with email alerting at 90%, 95% and 100%

'Profiles' folder is shared, and subfolders for each user with permissions

 

I'd like to be able to make use of our existing user-creation tool (tools4ever User Management Resource Administrator, not cheap but works brilliantly) if possible to create the shares and permissions, but obviously there's no way for it to create shares or set root-level and share-levels ACLs on the Sun box. I know the Sun OS will take scripts, so that's definitely a possibility. Whatever solution we use needs to be manageable by everyone in my department, who are coming from a mainly Windows-only background.

 

The easy alternative is to have a Windows 'head' with thin-provisioned storage attached via iSCSI then we just manage everything via the Windows box. It's not ideal as we have another bottleneck and point of failure, but it does make everything simple in a Windows environment!

 

 

Butuz - I've got two Xen boxes here at the moment and have been very impressed! I couldn't believe how easy it was to set up compared to Hyper-V. The fact HA just takes a central heartbeat and a tickbox compared to having to completely configure Windows Clustering is brilliant. My only concern is that Xen hasn't got anywhere near as much market share as VMware and the features aren't quite there yet (check out the VMware 2009 roadmap and watch the Fault Tolerance video). I've yet to give ESX a proper test so I'll see how it goes.

 

It sounds like your plan is very similar to ours in principal so I'd be interested to know how yours goes!

 

Cheers,

Chris

[Ric_]

@Duke: You should still be able to create your homedrives, etc. as you do now. The only difference will be that you won't create a username$ share for each user - simply a users share with subfolders.

 

There is a script on the wiki that will sort your permissions out (it uses XCACLS) once you have connected to the relevant share.

 

Profiles can be done in the same way - Profiles share, subfolders and script the ACLs.

 

The only thing that I haven't sussed yet is how to take advantage of quotas. I have time to figure this out though because of the jump in storage space that I am taking.

 

I use XenServer too. I'm not sure exactly which features you are after but the main ones are there already and it is still the cheapest option for lice migration of VMs. The HA is also awesomely simple to use as you mention.

 

If you want to have a chat about things, please give me a shout and try to make it to the conference (although Andy mentioned you had some stuff on to do with the provisioning of the beast).

[Duke]

Hi Ric,

 

Thanks for your reply, much appreciated. I did a quick test with a 'users' share and a 'profiles' share but had some issues with permissions. Even though through Windows their effective permissions were full control, they still didn't have permission to access \\sunbox\users$\username, but I'm sure this was down to me getting something wrong! If we can get this working then our user creation tool should still be able to create all the userspace and profile folders for each user (around 250+ each year) which will be a big bonus.

 

We would be okay on the quota front too if required as we'll have way more space than we need. However, with 1600 users the potential for them to fill up their userspaces with large files is worrying, so it's nice to have a cap on them. Not sure if you've ever played with File Server Resource Manager but it's actually a really good tool, but obviously limited to the local Windows server it's on. If Sun could find a way to create something like 'Set quotas on all folders in the top level of this share at: 300MB' rather than just a quota for the share that'd be brilliant.

 

We're looking at getting Phil and someone from Cutter, plus a colleague at a local school in at the end of the month, so I'm sure we'll be able to figure out a better idea of things then.

 

Cheers,

Chris

[Butuz]

Where are you based Chris?

 

Butuz

[Duke]

In Wiltshire - Hardenhuish School.

 

Chris

[Ricko]

Hi Chris,

 

I've been testing tools4ever today using the 7000 series simulator.

It works fine to create user folders within a share set up with the Sun web interface.

 

I would create a profiles share and perhaps a userspace share for each year group. That way the tools4ever script can create all the user folders within the shares and can inherit permissions from the Root ACL. You could then do as Ric suggested and use something like Robocopy to move the existing user folders to the new shares.

This does not help with the quota issue though. I have sent you an email with details of the permissions I used during my testing. I hope it's of some use to you!

 

Rick

[Hebdenlad]

Hi folks,

 

First post, but I've been keeping an eye on the forums as a couple of people I've been working with are on here. I've recently bought a Sun 7410 Unified Storage System with a 22TB J4400 Array and three flash accelerators. I know there has been quite a bit of discussion on the Sun 7000 series, but it's been mainly aimed at the lower end 7110 and 7210 models, so if anyone has any questions about the 7410 then I'll do my best to answer them, although be aware I haven't fully deployed it yet in production so I won't have all the answers!

 

(Thanks to Andy at Cutter Project and Phil at Sun for their help, very good people to work with!)

 

On to my related question...

 

For those of you who have a SAN and Windows users with roaming userspaces/profiles, how have you got them saving directly to the SAN? At present we have three Windows servers with shares for userspaces and profiles, these locations are set in the user account in Active Directory and the 'My Documents' folder is redirected with Group Policy.

 

I'd like to have these users (1600+ of them) writing directly to the SAN to remove the Windows server overhead and point of failure. However, I'm starting to realise some issues like the ability to batch-create user shares and set permissions on them (easily done with our Windows user account tool), granular control of file and folder permissions, and quota management at the level provided my Microsoft's File Server Resource Manager.

 

If you're using a SAN with lots of Windows users, how do you create and manage the shares required on a large scale? I realise this isn't a problem with the Sun box (it fully supports AD and CIFS, and does no less than NetApp or EMC kit to my knowledge), but more a 'what's the best practice' question.

 

Many thanks,

Chris

Hi there "Duke",

 

more than happy to help on this one if I can?

 

can you contact me so we can start talking?

 

I know some of the guys on here have roaming profiles running on their S7000's

 

Cheers,

 

Phil

[torledo]

Hi Soulfish,

 

I'd previously looked at NetApp, EMC, Dell, SANMelody and others for storage solutions, but at the promo pricing the Sun box was very hard to argue with. I already have a 4TB NetApp box so was seriously considering going with NetApp, but the Sun 7410 was basically half the price of a comparable NetApp 2050, plus all the licensing and OS is free!

 

Cheers,

Chris

 

 

hi chris,

 

can i ask which netapp box you currently have ? A fas 250 or a fas 270 is it ?

 

had you basically maxed out the storage capacity on your netapp, also which feature licenses are you running ie snaprestore ?

[Ric_]

@Duke: Have you upgraded the 'firmware' image on the box? The latest release(s) have fixed some Windows ACL problems.

[teejay]

OK, got roaming profiles and users set up and live on our Sun 7110.

This is the easiest way of doing it:

 

Make sure you have the latest update on your Sun box, makes this work a whole lot easier!

Create a Project for the Users, eg Students.

Create a share for each group of users, for instance for each Year Group.

On the CIFS share level ACL set your admin group to full control and give a security group with the users who are going to have their home directories in this sgare all read and write access but don't tike the inheritance options.

In the CIFS Root directory ACL give your admin group full control and the share users group:

Read Data/List Directory

Execute File/Traverse Directory

Read Attributes

Read Extended Attributes

 

Then, in the share create two folders, Documents and Profiles

On each of these give your Admin group Full Access

Your share users group:

Traverse Folders/Execute File

List Folder/Read Data

Read Attributes

Read Extended Attributes

Create Folders/Append Data

and set this to apply onto this folder only

Add permissions for CREATOR OWNER as everything apart from Change Permissions and Take Ownership and set this to apply onto Subfolders and Files only.

 

Right, to copy your users across, use:

robocopy \\oldserver\share \\sunbox\share\documents /E /ZB /DCOPY:T /COPY:DATO /W:1 /R:1

 

where \\oldserver\share is the location of your users Documents folders. You can do the same for the profiles if you wish, but I found it best to let the profiles be recreated. You may get a few users complaining about missing favourites etc though!

 

Couple of caveats on this, first you need to make sure the user is the owner of their Documents folder and their profile folder.

 

I used /COPY:DATO rather than COPYALL as we found trying to copy across the ACL from the old server wasn't very successful. Using /COPY:DATO just copies across the ownership and the other permissions are inherited from the folder you are copying into.

 

Hope this helps! We have now copied across 4 year groups without any problems using this method.

 

PS - I'm extremely jealous of your 7410 setup!

Edited May 11, 2009 by Ric_
Removed stray smilies :D

[Duke]

I've been testing tools4ever today using the 7000 series simulator.

It works fine to create user folders within a share set up with the Sun web interface.

 

Rick - This is great, and thanks very much for your email! I'll have a play with it later on and do some testing. It's a shame about the quotas, but I fully appreciate this is simply because we've become used to relying on FSRM on a Windows box, not because of any limits of the hardware. I think that right now making it easy to set up the folders and permissions is a bigger issue than quotas.

 

can i ask which netapp box you currently have ? A fas 250 or a fas 270 is it ?

had you basically maxed out the storage capacity on your netapp, also which feature licenses are you running ie snaprestore ?

 

torledo - We've got a FAS2020 so it's the bottom end, rather than a higher-spec 2050. There was nothing wrong with it in any way, but it's used primarily as our D2D backup box. I got a very good end-of-year deal on it last year so snapped it up, but the problem I have is that we struggle with their prices in general. I had a quote from a NetApp supplier for the whole virtualisation and storage project, and while it was very nice, it was also very expensive! I couldn't justify tying the school into a provider that we could only just afford each time, even with a discount, as it would be unmaintainable in the long term. I still think of NetApp as a top-tier provider, but when you compare the Sun prices and features (Fishworks BUI, D-Trace, etc) then I think they're getting ever closer.

 

Licenses are something like Nearstore, Snaprestore, A-SIS (limited to 500GB, absolutely mad), CIFS and iSCSI. Knowing that all of these features are provided free by Sun, and that any future upgrades will be no extra cost is a big plus. I know the whole open-source open-storage thing can be over-hyped, but in this case I think it's a real plus.

 

@Duke: Have you upgraded the 'firmware' image on the box? The latest release(s) have fixed some Windows ACL problems.

 

Ric_ - Yep, I'm sure the initial permissions problem I had was down to me, I just need to set up the share and root level ACLs properly. Still in the early days of testing and figuring out best practices!

 

OK, got roaming profiles and users set up and live on our Sun 7110.

This is the easiest way of doing it [...]

 

teejay - Awesome mate, cheers! I'm going to go through your post and Rick's emails and I think I'll be in business! As your setup sounds very similar to our requirements, how are you handling quotas, or do you not find it to be an issue?

 

Thanks all!

Chris

[teejay]

You can put a quota on the share, but not individual folders. Maybe one to nag Phil about!

 

TBH, we don't bother. If they start eating into the space, then we start checking into things, but with the amount of media based courses we now run then quotas are just a pain.

 

The advantage of sticking them all in one project is that you can set up the snapshots on the project, which is a very usefull feature, integrates extremely well with the Windows previous versions client and doesn't have random hissy fits like Windows snapshot tend to (for us anyway).

[Hebdenlad]

You can put a quota on the share, but not individual folders. Maybe one to nag Phil about!

 

TBH, we don't bother. If they start eating into the space, then we start checking into things, but with the amount of media based courses we now run then quotas are just a pain.

 

The advantage of sticking them all in one project is that you can set up the snapshots on the project, which is a very usefull feature, integrates extremely well with the Windows previous versions client and doesn't have random hissy fits like Windows snapshot tend to (for us anyway).

Thanks for the mention. Regarding the quotas - all will be revealed tomorrow for you Trevor. Other guys - see my posting about the NDA presentation. you'll then see what is happening with the S7000 series :-)

 

Cheers,

 

Phil

[Duke]

teejay - It'd be cool if Sun could consider doing something like that for quotas (e.g. apply to subfolders, not the share itself). It's only really needed for something like userspaces, but I would imagine CIFS users would be a fairly common use for these boxes, particularly in education.

 

Thanks for the reply,

Chris

 

EDIT - Phil, can't wait until the end of the month!

[torledo]

Rick

 

 

torledo - We've got a FAS2020 so it's the bottom end, rather than a higher-spec 2050. There was nothing wrong with it in any way, but it's used primarily as our D2D backup box. I got a very good end-of-year deal on it last year so snapped it up, but the problem I have is that we struggle with their prices in general. I had a quote from a NetApp supplier for the whole virtualisation and storage project, and while it was very nice, it was also very expensive! I couldn't justify tying the school into a provider that we could only just afford each time, even with a discount, as it would be unmaintainable in the long term. I still think of NetApp as a top-tier provider, but when you compare the Sun prices and features (Fishworks BUI, D-Trace, etc) then I think they're getting ever closer.

 

Licenses are something like Nearstore, Snaprestore, A-SIS (limited to 500GB, absolutely mad), CIFS and iSCSI. Knowing that all of these features are provided free by Sun, and that any future upgrades will be no extra cost is a big plus. I know the whole open-source open-storage thing can be over-hyped, but in this case I think it's a real plus.

 

Chris

 

chris

 

thanks for the info. the 2050 isn't really that much higher spec than the 2020. What it offers is expansion of ports via a single PCI card per controller. This would allow you to say bump up the number of ethernet ports for NAS or iscsi by four ports per controller. The 2020 is restricted to just 2 ports per controller.

 

The 2050 also accomodates 8 more disks in the base shelf - so 20 disks vs 12, and apart from a memory boost it's functionally on a par with the 2020. I understand your reservations for scaling such devices. Adding additional disks to a 2020/2050 isn't cheap!!! if i was buying netapp or the ibm n series [oem'ed from netapp] i'd buy everythign i needed up front and take advantage of discounts that way....adding later is unlikely to provide the same sort of discounts.

 

I agree with you about the licensing aspect, but i don't really think you need licences beyond snaprestore for your environment. NFS might be handy but it is pricey as are features like flexclones. But the a-sis dedupe feature is now available when the unit is used as primary storage at no additional cost. No need to convert it to a nearstore appliance [which is also free but...], the nearstore appliance actually requires additional licensing in the form of snapvault secondary storage licensing to be used as a true D2D appliance. It's pretty needless expense unless you have two filers in which case you can use the very cool snapmirror/snapvault combo. Yet again, it's expensive.

 

hopefully what sun have done will encourage the likes of netapp and emc to review the pricing for their low end kit. Most of this stuff netapp charge really should be free. And the price difference between a single controller vs dual controller netapp box with CFO licensing is insane imho. dual controlelrs should be as standard and a lower entry price point.

 

now that you've got your 7140s the 2020 is handy as a d2d box, but i think it would work really nice as primary storage somewhere in the org. If you've got 4TB in the 2020 why not use it for your file serving needs ? and keep an eye on the enhancements to single instance storage features of the box.

sis for primary storage is relatively new still.

 

if you've got dual controllers i'd definitely try and use the 2020 as primary storage.

[Duke]

Hi torledo, thanks for your reply.

 

The plan is eventually to get another 7410 and mirror them across the site. As the 7410 has four Ethernet sockets and can easily be expanded to more (the FAS2020 is just a single controller unfortunately, couldn't afford a dual one), plus has the performance boost from the Flash accelerators and is 22TB rather than just 4TB, I think it makes the most sense for it to be our primary storage unit.

 

I completely appreciate your point about making better use of it though. We needed D2D backup that would offer us iSCSI and CIFS, and I got a really good deal on the price so it made sense at the time. We purchased it with the full knowledge that we wouldn't be making full use of all its features for a while.

 

As you say, a lot of the very expensive stuff from NetApp should/could be free, and they'd probably get a lot more lower-end customers. Regarding NFS - it's been suggested NFS will get better performance than iSCSI for use with VMware virtual machines in ESX, anyone have any feedback on that?

 

Cheers,

Chris

[Ricko]

Hi Chris,

 

From what I have read on the 7410 performance testing blog, NFS seems to be considered the fastest protocol available on the 7410. This would go for any use including ESXi storage.

 

Rick

[Duke]

Thanks Rick, I'll have to give NFS a go. Once I get some time (unlikely at this rate!) I'll set up a couple of ESX servers and do some proper testing. I've always leaned towards VMware as it's the more established product, but it's nice to see a lot of people going with Xen as it seems so simple to use...

 

Cheers,

Chris

[linescanner]

The other advantage in using NFS is that the analytics you can get from the system is better than iSCSI

[Duke]

And we all know how much everyone loves the D-Trace graphs!

 

I'm off to have a play with the 7410, cheers guys!