Oracle ZFS Storage Appliance (S7000) Q3.3.1 Released

[RobFuller]

@Butuz which version of firmware was that box running out of interest if you know?

[linescanner]

@linescanner I think we all know the escalation process all too well... it doesn't necessarily mean that your box gets fixed double-quick though. The speed of response really doesn't help when their answer is 'well, you're just going to have to wait until it finishes what it's doing' or 'yeah, you shouldn't turn that feature on'.

As people know, I've been a long standing Sun customer and all my server estate is Sun kit... plus I've had workstations and Sun Rays. Recently though I've lost confidence in the Oracle brand. I've told them this and they don't even seem that bothered.

Agreed and the points have been raised. We are on your side and are trying our hardest to get a suitable response. I now have twice monthly calls in place to air some of you guys issues etc.

[Duke]

I hate to be the odd one out here, but having worked with Andy T (Cutter), Andy P (WTL) and Phil L (Oracle) for a while on the S7000 stuff, I can say our S7410 has generally worked very well, been well supported, and been a product I'm happy I bought. I think there are a lot of happy S7000 customers out there, but generally people only post on forums when something is broken rather than when things are working perfectly. I'm sure there are a lot of S7000's out there working well, but it looks like the original S7110's and S7210's haven't been as well supported as the S7310/S7410 + S7x20 range. That's definitely not okay by any means, but it doesn't mean the whole product range is completely cr*p. My NetApp FAS2020 sucks, but NetApp's higher-end stuff is successfully used by huge organisations around the world.

I hope I don't get hated on for saying this, but I still stand by the S7410 being a good product and I assume the S7x20 stuff is too. The Fishworks OS, DTrace, and the protocol licensing model are still all excellent innovations that most other storage providers are still playing catch-up to. I spoke to Phil L yesterday (sorry again for scaring you Phil) and I know he and Andy from Cutter have worked with Steve Darkes (Oracle - good guy from everything I've seen) to really try to get things moving on the issues people here are having. I just hope Oracle can turn things around because we had a great community on here around the S7000 stuff and I'm not sure we'd get the same 'personal' level of support we got from Phil Lawrence from a company like EMC or NetApp.

Cheers,
Chris

[linescanner]

Thanks Chris. I just want to echo this, I have S7000 series customers including 7110 users that are having no issues at all. Steve Darkes (Oracle) had a long chat with me yesterday and is working hard to sort things out.

Will keep you all posted

[linescanner]

Have had response from Oracle. Will update in a bit.

[john]

I hate to be the odd one out here, but having worked with Andy T (Cutter), Andy P (WTL) and Phil L (Oracle) for a while on the S7000 stuff, I can say our S7410 has generally worked very well, been well supported, and been a product I'm happy I bought. I think there are a lot of happy S7000 customers out there, but generally people only post on forums when something is broken rather than when things are working perfectly. I'm sure there are a lot of S7000's out there working well, but it looks like the original S7110's and S7210's haven't been as well supported as the S7310/S7410 + S7x20 range. That's definitely not okay by any means, but it doesn't mean the whole product range is completely cr*p. My NetApp FAS2020 sucks, but NetApp's higher-end stuff is successfully used by huge organisations around the world.

I hope I don't get hated on for saying this, but I still stand by the S7410 being a good product and I assume the S7x20 stuff is too. The Fishworks OS, DTrace, and the protocol licensing model are still all excellent innovations that most other storage providers are still playing catch-up to. I spoke to Phil L yesterday (sorry again for scaring you Phil) and I know he and Andy from Cutter have worked with Steve Darkes (Oracle - good guy from everything I've seen) to really try to get things moving on the issues people here are having. I just hope Oracle can turn things around because we had a great community on here around the S7000 stuff and I'm not sure we'd get the same 'personal' level of support we got from Phil Lawrence from a company like EMC or NetApp.

Cheers,
Chris

I think the issue has been that Sun / Oracle gave the moon on a stick and said use it and enjoy the moon on a stick which we have and then after enjoying it, the moon wants to go back to being many miles away from me.


Both myself and Ric have pushed limits with our boxes, we have used all protocols, services and features, as they were sold as boxes to do all that stuff, as the stuff under the hood has got fancier they have failed to keep going and the hardware has given out. Regularly the 7110 has DRam issues in that its not got enough, yet if I double it I bet some of my issues will go away /reduce as will my Oracle support (well likely to have arguments bit like top PC makes and non-genuine parts). Other issues are still occuring with mine but I bet if I double the DRam it will improve an number of issues that we are presently having with it. The analytics are the best feature and I love that as I can see whats going on and show the kids what is going on visually when they all try and load HD Video at once etc... so on that score 10/10

I think the high end S7*00 range is probably better suited due to the tin it runs on that is what is holding it back I think now, its like running Vista on a "vista ready" PC it never was going to work with a lot of them, I see them now as "S7000 ready" boxes when actually they are not fit for it any longer without dual processors and more DRam.

The S7120 we have is working, its got issues but not on as grand scale as the 7110 thankfully (touches lots of wood!!!) buts its not without fault I'm afraid and that has more grunt, flash accelerators, more ram etc....

I have been dealing with Steve and others at Oracle on my issues and shouted about the Escalation process and it sort of works, my main issue is still a P1 case yet updates are not fast enough for my liking, however we do have to slow it down at times waiting for a logable incident of our problems to collect / provide data so I accept that we dont' always help keep it flowing fast but reply#s once we provide the data I would expect to be very prompt.

Once again I want to say that @linescanner (Andy) and @kmount (Kim) from Cutter have been brilliant throughout this saga and still are supporting me with it so full marks to them as a reseller, I owe them more beers and dinner for all the help and support they provide with this. Phil from Oracle is also very helpful and he has been poking and kicking people for me as well

[linescanner]

Chaps,

as promised an update from Oracle.

Service Request Handling

1.Provide root cause where possible for each open Service Request - with workaround, hardware or firmware fix, or interim binary relief to resolve
2.Where not possible to establish true root cause, provide probable or possible cause(s), and data-gathering action plan to confirm upon next occurrence
3.Provide clear escalation procedures when SR response does not meet expectation, or when management awareness is needed

I am aware that some of you are experiencing long gaps between updates from the support engineer involved, should this be the case you need to provide an idea to the engineer of what level of response you require ie (once a day even if to say we are still working on it)

I have also raised the fact that some of the engineers assume that the end user is happy in a Solaris console and are asking end users to run commands from here etc. I know some of you chaps are *nix friendly but I am also aware that a load are not. This has been passed on to Steve Darkes to address internally.

For all you looking at this thread and thinking......eeek....lastest upgrade is looking dodgy/scary read on

Q3.3.1 Upgrades

1.Q3.3.1 upgrade process includes new health-checks with aim to reduce failure during upgrade attempt (specific examples : SIM path offline, SSD path offline)
2.Customers concerned regarding Q3.3.1 upgrade process should raise a Service Request, requesting pre-upgrade checks (log file size in particular)
3.I will provide you with a list of the Bug fixes already implemented to improve log file management, to demonstrate improvement has been made
4.Log file issue specifically observed by Settle College is being root caused by the Oracle TSC engineer who restored this customer's system

As soon as I have the details on note 3 here I will pass them on as it is something that concerns us also.

Make use of note 2

S7110 Performance Constraints

Cutter will work with Oracle to provide feedback to Scott Meadows (ZFSSA engineering manager)
on issues arising from memory pressure on low-end (S7110) through use of specific features:

(ie deduplication), and through lower potential I/O headroom potentially leading to more severe relative performance degradation under high load.

Regular Service Review

1. Bi-weekly call agreed, initial run of 4 calls - to bring, discuss and respond to technical issues and concerns brought by Cutter Project customers
2. Email follow-up to be provided with customer-ready response to help explain and where necessary counter, feedback originating from individual customer issues

So to that end, if you are one of my customers and you want to have us raise anything about the above, please contact me. Oracle do genuinely seem want to get this resolved as are we. I have lots of these boxes out with customers and they underpin our VDI offering.

And if it all goes wrong.........shout at Phil

[Duke]

So, for the sake of balance I figured it wouldn't hurt to post my recent Oracle support experience:

On Monday we had a disk failure on the 7410 but it ended up being a slightly larger issue:

(pool is RAID-6 over 20 disks + 2x hotspares)

• Disk 2 failed
• Hotspare 23 kicked in
• Resilvering started
• Resilvering completed
• I replaced Disk 2 with one Oracle sent me NBD
• Resilvering started
• Disk 2 failed (the new one)
• Hotspare 3 kicked in
• Resilvering completed
• Both hotspares are still active

At this point my first-level Oracle support guy was confused, as I essentially had a 20-disk pool running over 21 disks. He passed my case to the ‘recovery’ team (they need a better name that doesn't scare the out of customers). I added a note to the case this morning asking if they could ship a replacement disk (as I assumed I'd need one anyway) and could I raise the priority of the case as it had gone from just a simple disk replacement to something more serious.

The second-level Oracle guy phoned me back in 5 minutes, initiated a shared shell and fixed the problem in another 5 minutes with no downtime. It turned out to be a software rather than hardware bug (the second disk failure - the first one was actually a dead disk). The bad news is that it was a known issue, and they hadn't managed to get the fix built into the latest release. The good news is that (fingers crossed) it's fixed with no downtime. So, overall not ideal, but not too shabby either.

Chris

[john]

Nice to hear of a good resolution Chris

We're still playing ping-pong on ours But the complaints are stacking up internally now, if I wasn't off to the F1 this weekend I would be finishing the data migration off it, but its past the point of acceptable / workaround / blagable now so I'm going to have to finish the migration and bin the system and start again on it

I hear back from Oracle every couple of days about it saying we are still working on it so I am sure they am but there not the one with the queue of staff and students who have "lost" work due to it mapping but not mapping etc

[apaton]

Didn't really want to start another thread, but yet another update is now available 2010.Q3.4.0.ak-2010.08.17.4.0 Release Notes - Fishworks - wikis.sun.com

Andy

[Duke]

Didn't really want to start another thread, but yet another update is now available 2010.Q3.4.0.ak-2010.08.17.4.0 Release Notes - Fishworks - wikis.sun.com

Whaaaat? I just upgraded to 3.2.1 so we can do a BIOS upgrade then go to 3.3.1 on Monday, looks like that plan just changed! From what I can see I can go straight from 3.2.1, but I hate upgrading to stuff that's only just come out.

EDIT: Ooh, fix for a bug I hit though: 6999699 - Two spares disks used for the same failed disk and not detached after replacement and resilvering

[teejay]

Ooops, I'd completely forgotten about updating our S7000's, will add it to my job list.

[Duke]

Has anyone here done a BIOS update on an S7000? Apparently the Fishworks updates don't include them and they have to send an engineer to manually do it. The guy dealing with my support case has said we should do one prior to upgrading to the latest release, but so far he hasn't filled me with confidence, and I'm hoping the engineer they send has some experience doing this...

In unrelated (but awesome) news:



Hehehehe!

[kmount]

(I've done some...), it's mostly about marrying up settings in the BIOS afterwards.

If you want to, ping me the name of the engineer you're having visit and I'll see if I know of him

Moreover, ping me the name of the one you're dealing with if you're concerned to see if I know of him too. (we can also raise awareness of this with the SR in our escalation process)

Or drop @Hebdenlad a message to sound stuff out by him.

[Hebdenlad]

Thanks Andy!

Seriously though. If anyone has issues then please, please contact me either on here or via my Oracle email address (philip.lawrence@oracle.com). The only way we can fix your issues and address your concerns is if we hear back from you so please call or email me if you have issues or concerns.

WR

Phil