S7000 NFS with ESXi

[Duke]

This has been kind of covered before but never definitively - just wondered if someone could sanity-check my NFS setup and whether we could put together a best-practices guide for uisng NFS and VMware on the S7000?

 

My setup - currently overly paranoid but I want to get it right:

• Dedicated storage switch connected to SAN and ESXi host
  
• ESXi host has dedicated storage NIC on different IP range to rest of network
  
• SAN NIC port to be used for this is on same IP range as ESXi storage NIC
  
• Made a new project for ESXi, left all the defaults but turned off update access time on read
  
• Make a new share and set NFS Share Mode access to none
  
• Set up a read/write NFS exception of the ESXi IP as x.x.x.x/30
  

 

What should the owner and group be set to for the share, or does that just apply to CIFS?

Does the NFS exception need root access (past tests would suggest no)?

Who does the ESXi box connect to the SAN as and how is this authenticated?

 

Other resources:

Using Sun Storage 7000 Unified Storage System With VMware ESX Server

 

Cheers!

Chris

[Duke]

Okay, so THIS thread answers most of my questions (I thought it was on here somewhere, couldn't find it though!).

 

Is Andy's final comment the recommended way of securing NFS ACLs for ESX access?

 

1) Create a share with access control of "nobody" and "other"

 

2) Under NFS Exceptions select "ROOT ACCESS" check box for specific host

 

Thanks in advance,

Chris

[apaton]

Yes!

 

Best practice should also include/consider

 

Create Projects based on VM replication requirements.

Which VM's are in which projects

Snapshots usage (if used at all / need to consider VSS etc..)

Write Policy - Latency/Throughput

Access time (atime)

 

VMware ESX NFS tuning (Advance settings)

Carefully select which NIC's have a Kernel Port. (I have seen in ESXi traffic going over the wrong NIC a few time, as the default network port is a Kernel port)

 

Network Topology - Redundant Switches/802.ad/Jumbo Frames etc..

 

Andy

[Duke]

You're a legend sir, I shall have a play.

 

Quick question: The VMkernel port that's created by default is described as a Management Network. Is this required to connect to the ESXi box via the vSphere Client? Just wondered if I could remove VMkernel from the curriculum network vSwitch completely to avoid the risk of any storage traffic attempting to go through the main network rather than the storage. I assume that's not possible...

Edited December 9, 2009 by Duke

[Duke]

Well that's been a big old fail...

 

ESXi host has two single-port NICs:

 

• One connected to the main network so that the VMs can be accessed by clients and so that I can manage the ESXi host - IP is on the main network's IP range
  
• One connected to the storage network to handle NFS traffic - IP is on the storage network's IP range
  

 

Obviously the SAN is also connected to both the main network (main IP range) and the storage network (storage IP range) as in some cases it's being access directly by clients for CIFS and in some cases I want to use the storage network.

 

Since both the ESXi host and the SAN are connected to the main network, ESXi is just ignoring the storage network completely and using the main network and its associated NIC. My first thought was that I'd been stupid and when I set the NFS exception I'd used the ESXi hostname (which is on the main network). However, I changed this to the.storage.ip.range/24, and now I can't connect the ESXi host to that NFS share.

 

I've obviously missed something as I know other people are doing this, any suggestions?

 

Chris

[Theblacksheep]

What are the port groups?

 

you should be looking at, something along the lines of

 

vSwitch0: Service Console: 'curric' (vswif0) & VM traffic

vSwitch1: Service Console: 'storage' (vswif1) & VMKernel (vmk0)

 

Have you got rid of any pre-configured shares and readded them?

[Duke]

That's what I figured, thanks for the sanity check...

 

Here's what I've got

 

vSwitch0: VM Port Group for VM Network & VMkernel Port for Management Network (curric IP)

vSwitch1: VMkernel Port for storage (storage IP)

 

Obvious solution would be to remove the VMkernel port from vSwitch0, but if I do that how do I connect vSphere to the ESXi host to manage it?!

 

(bearing in mind I'm running vSphere Client from my desktop PC which is on the curric network with a curric IP and not connected to the storage network)

 

I have no service console as such, running ESXi 4.0 U1.

 

Chris

[Duke]

Panic over, I'm a idiot...

 

Everything was configured fine on the NICs and vSwitches, the problem was that when I added the NFS datastore to the ESXi host I was specifying the SAN using its DNS name on the curric network. Thus, when ESXi tried to connect to it, it used the curric vSwitch to talk to it.

 

I've just added the NFS datastore by specifying the SAN as the storage network IP, and all of a sudden it works perfectly and NFS traffic on the SAN is showing up as coming from the storage-IP on the SAN.

 

Completely my fault!

 

Thanks for the help guys...

 

Chris

Edited December 9, 2009 by Duke

[Duke]

Holy hell!

 

NFS

IOPS: 2076.52

MBPS: 22.04

Avg Response: 0.4803

Max Response: 366.86

CPU Util: 20.22

 

Why didn't I do this sooner?!

[Theblacksheep]

Holy hell!

 

NFS

IOPS: 2076.52

MBPS: 22.04

Avg Response: 0.4803

Max Response: 366.86

CPU Util: 20.22

 

Why didn't I do this sooner?!

 

Nice, yeah its pretty good.... I was gonna setup a budget san next year with NFS.

 

Glad you got it sorted. Just to cheer you up, ESXi4U1a is out:

 

ESX(i) 4.0 Update 1A » Yellow Bricks

Edited December 10, 2009 by Theblacksheep

[Duke]

Looks like it's time to test the host update utility for the first time, thanks!

[linescanner]

Holy hell!

 

NFS

IOPS: 2076.52

MBPS: 22.04

Avg Response: 0.4803

Max Response: 366.86

CPU Util: 20.22

 

Why didn't I do this sooner?!

 

Don't say we did not tell ya