Sun Storage 7110 networking woes

[DMcCoy]

@ DMcCoy, this is what's confusing me I have used the CLi to enable it but it still doesn't show in the interface. My connection to the SAN is now working so it must be ok right???

 

This is the command I used and it seemed to do the trick.

2910(config)# trunk ethernet 2-4 trk1 lacp

 

 

 

I think you probably are doing it the right way, it's been a while since I tried trunks with lacp, and OS X is a bit difficult.

 

I've been playing with LACP on an unused xserve (remotely ). So far I've only been able to lose communication with it after setting the team up and applying.

 

The automatic trunking is actually when LACP creates a dynamic trunk (Dyn1 on my switch). Dynamic is pretty useless as you can't assign vlans to it etc.

 

You will probably find LACP is enabled for those ports when you enable the trunk ("sh run" and look for lacp statements).

 

 

One thing everyone needs to remember is that when the trunk is created you lose all the current vlan assignments for those ports and need to tag the newly created trunk.

 

 

"Show Trunks" will show you the status of LACP, "Show vlans Trk1" will show which vlans it's a member of.

[spc-rocket]

Question: To those of you who are making a single 4Gb trunk, do you find you really need that bandwidth and is your network really supplying enough data to maximise it? I would have thought you would have had bottlenecks elsewhere unless the SAN is just talking to servers that have a lot of NICs.

 

Also, I remember reading somewhere that the BUI can put out quite a bit of traffic itself when you're doing a lot of analysis, would it not be better to have a 1Gb link dedicated for the BUI on one IP address that you know will always be available and a 3Gb link for data?

 

Final question - assuming some of you are using the SAN for virtualisation, shouldn't you have a separate connection on the NIC for NFS/iSCSI/VMotion traffic on a different VLAN/IP range?

 

Sorry for all the questions, just trying to figure out how I'm going to hook mine up.

 

Chris

 

I think the 4Gb of bandwidth would be enought for a lot of applications and you are going to have to do serious hammering to saturate the bandwidth. Personally i think if you have 4 x1Gb ports on the san team two of the ports to the switch, use one for management and one for connecting to the main network for direct CIFS/NAS access.

 

Looking at the diagram that has been posted it seems that you are not seggregating your main network i.e. where your users are from the SAN. It is advisable to keep the san filer and the connection (iscsi) on a seperate network to the main network. I.e. have a specific switch to handle the traffic from the SAN filer to the servers and other switches that connect to various cabinets throughtout the establishment.

 

I would advise that people do a bit of traffic analysis i.e. how much traffic is actually used before purchasing extra NICs etc. Teaming more ports does use CPU on the file and also on the server end as well even if you have TOE card so make sure that people are aware of this.

 

I would advise the use of jumbo frames on the connections from the server to the san filer but not on the connections from your server(s) to the main network. Jumbo frames can make a hell of difference depending what traffic it is and how much data is flowing around. It also helps to reduce the CPU cycles as well.

 

Regarding the LACP configuration as other have pointed out make sure that both ends i.e. the SAN filer and also the switch are properly configured to handle or at least advertise that its capable of joining an LACP bundle/trunk. On cisco you would use the following command on each of the interfaces:

 

channel-group mode active i.e. channel-group 1 mode active

 

The first interface you will configure on the switch will create a virtual interface i.e. po1 and this will inhert the setting from the physical port. Not sure what HP is doing as cisco keeps the vlan info etc. Also trunk in cisco terminology means completely different - a link that carries multiple vlan traffic.

 

Ash.

Edited June 11, 2009 by spc-rocket

[Duke]

I would advise that people do a bit of traffic analysis i.e. how much traffic is actually used before purchasing extra NICs etc.

 

Definitely something I'll be doing once we start to deploy our 7410. I'm assuming most people here will have 1Gb or 2Gb links to their servers currently, so with the SAN you're already looking at having 2x-4x the network bandwidth. Our 'resources' file server is one box with a 2Gb link (no performance issues), but our userspaces/profiles are spread over three servers with 1Gb links. However, I've never seen the network maxed out on them so I don't see too much of an issue with having 1x 1Gb for management and 3x 1Gb for CIFS. We won't be doing virtualisation until next summer so I can buy another 4x 1Gb NIC then if needed.

 

Chris

[cookie_monster]

I've configured my storage as double parity raid and it says

 

Data 1.03T

Parity 300G

Reserved 16.7G

Spare 137G

 

what are the reserved and spare for can they not be used for data also, I need all the space that I can get.

 

 

It also says this when i'm configuring it but it doesn't show any issues in the problems page.

 

You have chosen to configure a chassis that is misconfigured or contains malfunctioning devices. If you configure storage using this configuration, you will not be able to add these devices without reconfiguring all storage. Are you sure you wish to do this?

Edited June 16, 2009 by cookie_monster

[teejay]

I've configured my storage as double parity raid and it says

 

Data 1.03T

Parity 300G

Reserved 16.7G

Spare 137G

 

what are the reserved and spare for can they not be used for data also, I need all the space that I can get.

 

 

It also says this when i'm configuring it but it doesn't show any issues in the problems page.

 

You should be getting 1.4T, looks like some of your drives aren't pickking up. Go to Maintenance, Hardware and under the image of the Sun box click show details, it should show you 16 disks and none of them should be offline.

[Duke]

How big are your disks? Spare is normally your hot-swaps (so a RAID rebuild can start immediately in the case of a drive failure), reserved is likely for system data or things like file tables. I'm not sure exactly how that works with ZFS. If it makes you feel any better I bought 22.5TB and I've got 14TB usable.

 

Not sure about the last bit regarding misconfigured devices though, that definitely doesn't sound healthy. Does anything stand out under Maintenance > Hardware if you got through all the devices? Are all the HDDs on the actual box lit up correctly and not showing an error? I had to re-seat one of ours when I fitted it.

 

Cheers,

Chris

[cookie_monster]

Oh yeh I have three missing disks, thought it didn't seem right. I think on the problems page it sould say do you know you have three missing disks.

[Duke]

Oh yeh I have three missing disks, thought it didn't seem right. I think on the problems page it sould say do you know you have three missing disks.

 

That's a pretty fair point, if it knows there is a chassis that is misconfigured or contains malfunctioning devices then that's kinda what I'd call a problem.

 

Chris

[cookie_monster]

Odd they were all lit up on the front of the device, I reseated them and they're all back now.

 

Cheers.

 

 

EDIT

 

That's more like it

 

Data 1.40T

Parity 316G

Reserved 22.8G

Spare 137G

Edited June 16, 2009 by cookie_monster

[teejay]

Odd they were all lit up on the front of the device, I reseated them and they're all back now.

 

Cheers.

 

 

EDIT

 

That's more like it

 

Data 1.40T

Parity 316G

Reserved 22.8G

Spare 137G

 

Your next job needs to be putting the latest firmware update on before you do anything else! Also, if you are using it with an HP switch, make sure that is also on the latest firmware.

After that, get the Lights out management working. To do this, make sure the lights out network socket is cabled and check in DHCP for an address with the dns name beginning with sun and then go onto that address with your web browser on an https session and login using your root username and password you set up.

[cookie_monster]

Latest firmware on the SAN is done and the latest firmware is on the HP 2910al switch, i've also got the ILOM working as well. I'm currently looking at an iSCSI LUN to present to my Xenserver box.

[sjl]

All this is very interesting stuff - we are just waiting for our shiny new x4140's and 7110 to be delivered.

 

We are using the two X4140's & 7110 with VMWARE to virtualise the SIMS server, print server one DC / DNS, a DHCP server, RIS / DEPLOY / Anti-virus server - Aprox 6 servers in total once its all set up.

 

On advice we will probably keep a the root DC physical.

 

We will be using CIFS shares so users can directly access them via the network

 

We'll have a private network for SAN traffic.

 

We will be using NFS instead of iSCSI for our VMWARE images.

 

We will present drives with iSCSI to the SIMS server for the SIMS DB etc

 

Teaming - we will trunk two of the ports on the 2 servers to the SAN switch then trunk together 3 of the links on the 7110. The one remaining link will go into our core switch (for the CIFS shares) as will one or two links (teamed) from the servers.

 

We are using a 1U LTO4 tape drive with BE11d without the NDMP agent for now. We will backup the CIFS shares to tape along with the SIMS DB (using the BE agent).

 

Not too sure what software to use to backup the VMWARE images i will need to trial some software but they will be going to disk then either archived to tape or removable storage - not too sure on this yet.

 

Those are the rough plans for now anyway and will probably change.

[cookie_monster]

We'll have a private network for SAN traffic.

 

Will that be a seperate IP range or just a VLAN?

 

 

We're in the same boat as you for backup probably use BE with remote agents to backup data but as for the VM's we might just have to copy the images to external HD's when we make alterations. We'll also have a physical DC.

 

We will be using NFS instead of iSCSI for our VMWARE images.

 

I've just had a play with an iSCSI setup between my Xenservers and the 7110 i'm going to see if you can use NFS with Xenserver tomorrow as it's supposedly slighty faster on the 7110.

[john]

FYI: BackupExec do VMWare agents for use with BE 12.5, specially designed to go into VMWare V Machines for backing them up at VMWare level and stuff

[DMcCoy]

FYI: BackupExec do VMWare agents for use with BE 12.5, specially designed to go into VMWare V Machines for backing them up at VMWare level and stuff

 

I have the licenses for this, but until I get my new servers in my BE box has not got direct access to the fibre SAN. You really have to want it at £1000+ per ESX box. You will also need a paid for version of ESX or ESXi (not that I know if it works with esxi) to get the VCB license.

[sjl]

Will that be a seperate IP range or just a VLAN?

 

 

We're in the same boat as you for backup probably use BE with remote agents to backup data but as for the VM's we might just have to copy the images to external HD's when we make alterations. We'll also have a physical DC.

 

 

 

I've just had a play with an iSCSI setup between my Xenservers and the 7110 i'm going to see if you can use NFS with Xenserver tomorrow as it's supposedly slighty faster on the 7110.

 

Its on its own SAN network - see V2 of our SAN diagram attached. Like you, once i get the servers and SAN i'll test NFS / iSCSI although it looks as though NFS is the better option.

 

Cheers,

 

Steve.

Edited June 17, 2009 by sjl

[cookie_monster]

Can anyone tell me if the management interface on the 7110 can be part of a bond?

 

I'd like 2 cards in a bond on a private network for iSCSI traffic and 2 cards onto the LAN but one of them will be the management interface, will that cause any issues?

 

 

EDIT: Found an answer to this.

 

Citrix recommends configuring management networks using a dedicated network interface or interfaces configured in a bond. The interface may be shared with VMs if desired; however, Citrix recommends separating VM, management, and storage traffic.

 

http://support.citrix.com/article/CTX118457

Edited July 7, 2009 by cookie_monster

[screwdo]

I've got my 7110 up and running but we've had also sorts of NFS timeouts. I've eventually got around the problem by hooking up our development server to the 7110 using a crossover cable. Sort of defeats the object of the NAS device, but it works and I'm happy.

 

Until recently I've been admin-ing the box via the iLO GUI and the nge0 GUI. All of a sudden I can no longer access the nge0 GUI. I can't even ping the NIC card from the dev server. I've reported the problem to the Network guys who want me to check the IP config and routing information. I've done this via the iLO command line.

 

start /SP/console > routing > select route-002 > show

 

This displays the type, family, destination, mask, gateway & interface. And all looks OK. The Network team are now asking me to test connectivity from the 7110 to the devices availble on the subnet.

 

My question is:- Is there a 'ping' type command that I can issue from the 7110 command line? Or is there any other way I can test connectivity from the 7110, so I can get the Network guys off my back?

 

Cheers

[Duke]

My question is:- Is there a 'ping' type command that I can issue from the 7110 command line? Or is there any other way I can test connectivity from the 7110, so I can get the Network guys off my back?

 

I'm pretty sure it'll do ping and trace route from the CLI. See here.

 

:> traceroute

error: usage "traceroute [host|addr]"

 

:> ping

error: usage "ping [host|addr]"

 

:> nslookup

error: usage "nslookup [host|addr]"

 

Chris

[Soulfish]

I've got my 7110 up and running but we've had also sorts of NFS timeouts. I've eventually got around the problem by hooking up our development server to the 7110 using a crossover cable. Sort of defeats the object of the NAS device, but it works and I'm happy.

 

Until recently I've been admin-ing the box via the iLO GUI and the nge0 GUI. All of a sudden I can no longer access the nge0 GUI. I can't even ping the NIC card from the dev server. I've reported the problem to the Network guys who want me to check the IP config and routing information. I've done this via the iLO command line.

 

start /SP/console > routing > select route-002 > show

 

This displays the type, family, destination, mask, gateway & interface. And all looks OK. The Network team are now asking me to test connectivity from the 7110 to the devices availble on the subnet.

 

My question is:- Is there a 'ping' type command that I can issue from the 7110 command line? Or is there any other way I can test connectivity from the 7110, so I can get the Network guys off my back?

 

Cheers

 

I've had NFS timeouts from our 7110 before. Issue we had was to do with NFS under opensolaris carrying out a reverse DNS lookup on the connecting IP and not being able to correctly resolve it to a hostname.

 

For us this only affected our storage network which was only being used for virtual machine images. Added a linux VM running bind with the correct rDNS and NFS now works perfectly