john Posted June 18, 2011 Posted June 18, 2011 Ok the 7120 is back alive after a few pokes. Credit to Rob the Oracle Engineer he has got it going, it seems these boxes are unable to look after themselves with log files growing and then taking ages for AKD to parse thus meaning service don't start, try to start in the wrong order and then jam up thus resulting in a bricked box as even a power on-off won't cure it as it will still read the files in the wrong order and jam up. So we have the 7120 alive again and on the latest release. He has kindly agreed to stay on and look at the 7110 and give it a check over for the log issue before we allow it to run the Q3.3.1 upgrade, and just like the 7120 it is log heavy and would likely suffer the same fate as the 7120 did which isn't that good as the end user cannot do these log things themselves thus clearly showing another failing of this hardware range. I would expect those with bigger disk arrays, more cache, flash etc to get by perhaps with the big logs as they will be able to cope better / faster I suspect but who knows. We shall see what this is like once it has the Q3.3.1 on it and if the work done in advance by Oracle's Rob keeps it away from the bricked box state that it usually gets into. 1
john Posted June 18, 2011 Posted June 18, 2011 Ok forgot to post when I finally got all the LAN back up and went home for a late lunch. Rob was excellent and we have both boxes on the new release, thankfully his log tidyin and sorting ensured a smooth upgrade, however its not a fault free box, chalk up SR number 5 that is now open for my Oracle boxes.... 110% credit to Rob at Oracle though and to Phil for ensuring we got them back up and working but they really do need to do something about the state of them, it seems some customers have no issues ever with them but some then have no end of issues and myself and @Ric_ seem to be the ones that have no end of suffering, probably due to us both using all the features and hammering the tin, but for what you pay for them you expect, want and need to hammer them to get the value out of them. We shall see if the ARP issue, dropping shares, analytic sheets crashing the box and other bugs cease with this new release, part of me doesn't expect it to fix them though, we still have the issue of shares deleted many many months ago still thinking they maybe there and it having a mind of its own thinking clients are accessing it when they are not doing so....
Duke Posted June 20, 2011 Posted June 20, 2011 Ok the 7120 is back alive after a few pokes. Credit to Rob the Oracle Engineer he has got it going, it seems these boxes are unable to look after themselves with log files growing and then taking ages for AKD to parse thus meaning service don't start, try to start in the wrong order and then jam up thus resulting in a bricked box as even a power on-off won't cure it as it will still read the files in the wrong order and jam up. I got hit by this once, scared the out of me when it wouldn't come back up after a reboot for an upgrade. I assumed they'd fixed this bug in a previous upgrade but apparently not.
john Posted June 20, 2011 Posted June 20, 2011 I got hit by this once, scared the out of me when it wouldn't come back up after a reboot for an upgrade. I assumed they'd fixed this bug in a previous upgrade but apparently not. Indeed not, it should be but not thus they had to manually clean all the logs up on both boxes, they did the logs for 7110 before we did the upgrade as it would have turned into a brick itself as well 1
Duke Posted June 20, 2011 Posted June 20, 2011 Might be worth asking if someone from Oracle will 'health check' my box before I upgrade. Got a busy summer this year and can't afford any unplanned downtime...
linescanner Posted June 22, 2011 Posted June 22, 2011 Indeed not, it should be but not thus they had to manually clean all the logs up on both boxes, they did the logs for 7110 before we did the upgrade as it would have turned into a brick itself as well Chaps, just to let you know we have not gone quiet on this. Am shouting to find out what, why, who etc. Will keep you posted. All Cutter customers that are having issues with 7000 series, please let me know ASAP. 3
john Posted June 22, 2011 Posted June 22, 2011 Thanks Andy for posting that, do let him know if any of you cutter customers are having issues please My saga continues, Oracle in there wisdom have decided that my 7110 has significant fragmentation within its file-system which is why it sucks, I have been reliably informed by two wise men today that ZFS doesn't fragment like NTFS! Thus clearly ZFS has some magical type of Fragmentation / its Oracle flannel to make me go oh yeh as a Windows guy I know about that and it can suck etc.... and 3 more bugs assigned to basically put my long-standing case on "hold" as it were until they release the next version of the software. Although quite what fragmentation has to do with 88% of my LAN traffic being Oracle based ARP requests astounds me?! The fact that the SAN's NIC isn't working at 100% and thus not overloaded means they are not interested in the rest of my LAN GRRRR For those considering the product my advise I am afraid is still don't touch them its really not worth it if you actually hammer and use the kit Shame really it worked great for a year / 18 months and just gone down hill since then. Full marks to Andy and Kim at Cutter, they have both been brilliant all through this and I am sure eventually we will get it right but until then steer clear of Oracle but do talk to Cutter about anything they can help with, I've NEVER used any vendor as helpful and supportive as them
Butuz Posted June 23, 2011 Posted June 23, 2011 Seeing posts like that John makes my decision (forced by Oracle I may add) to stick with my older SUN version of the firmware actually seem like not such a bad idea now! John could you not revert back to an earlier SUN version of firmware as an absoloute last resort? Butuz
john Posted June 26, 2011 Posted June 26, 2011 Seeing posts like that John makes my decision (forced by Oracle I may add) to stick with my older SUN version of the firmware actually seem like not such a bad idea now! John could you not revert back to an earlier SUN version of firmware as an absoloute last resort? Butuz Stick to Sun branded firmwares Thats my advice and I wish I could but too late now I've added all deffered updates so without binning it and starting fresh not much chance I am now working with a team of Oracle engineers around the globe including engineering guy's in the USA looking at it so it is being taken seriously just shame its taken so long to get to this point, I have migrated over half the data off it this weekend and making plans to do the rest next week as we need to be off it in my view as its no longer reliable which is a shame, but Oracle want to work with me to find the root cause and fix it but I need a job which is more key than them bug finding, the cases have been open since March in some cases thus they have had a large amount of time to dig around really, but we will do what we can with them.
RobFuller Posted June 28, 2011 Posted June 28, 2011 (edited) Oh heck ... I've been running my two 7110 for a couple of years now very happily (on the Sun firmware might I add) I haven’t upgraded in ages, the old saying if it ant broke don’t fix it. However I need to upgrade to a version that supports the hardware VSS provider for Windows so I can do SAN level VSS backups from my Hyper-V hosts. But after reading this scary thread I’m very worried about upgrading at all!!!! Does anyone know which version the hardware VSS provider is supported in or will I need to venture into Oracle scareware! Edited June 28, 2011 by RobFuller
apaton Posted June 28, 2011 Author Posted June 28, 2011 The latest VSS providers do need the latest Oracle firmware (Q3.3.1). There are few bugs in the old version.
john Posted June 28, 2011 Posted June 28, 2011 Ah.. this could be a scary upgrade path. I'd find another solution for your VSS, I really have no faith in the kit with the Oracle Firmware on it as I see that being the root of all evil tbh.
RobFuller Posted June 28, 2011 Posted June 28, 2011 (edited) Well its limited, I’m using DPM to do my backups on my VMs and it does work at the moment but I have to do serialised backups which take an age and do fail, if I can use the hardware VSS it could half my time but not at the expense of stability. Though to be fair I really only use these boxes as iscsi targets with replication nothing else! Worth the risk? Edited June 28, 2011 by RobFuller
john Posted June 28, 2011 Posted June 28, 2011 Well its limited, I’m using DPM to do my backups on my VMs and it does work at the moment but I have to do serialised backups which take an age and do fail, if I can use the hardware VSS it could half my time but not at the expense of stability. Though to be fair I really only use these boxes as iscsi targets with replication nothing else! Work the risk? Which ones are they that you have out of interest as I had a chat the other day with Oracle and we discussed the protocols and it was agreed that its good I don't use iSCSI as they are not suited for iSCSI due to no flash arrays in them like the new 7x20's have thus they are not performing as the should with iSCSI and nothing will improve that, as that was an early question about my performance did I use iSCSI to which i said nope I use NFS and SMB. I'd not gamble myself, although my issues seem to be more SMB related than NFS i have just no confidence, as soon as we get them to a working state they will never be upgraded again.
linescanner Posted June 29, 2011 Posted June 29, 2011 Guys, we do have customers running the latest firmware quite happily. I had a long chat yesterday with Oracle internal and am hoping to get some more info in a bit which I will post as soon as I have something. I know some of you have had a rough ride with the S7000 recently and that you have found support to be wanting, I have however spoken to the EMEA Unified Storage Manager responsible for support and stuff. He has been very helpful in explaining the process to make sure you are getting the best quality support and what to do if you are not. Below is a draft on the process and how to get the best out of it. 1. When should you escalate a Service Request ? Use the Oracle Global Customer Support Escalation Process when your business critical issue requires a higher level of attention from Oracle Global Customer Support Management. This process should be utilized when you: encounter a critical roadblock or showstopper to implementation or upgrade plans urgently need to communicate important business issues to managers in Oracle Global Customer Support are dissatisfied with the resolution or response to a Service Request 2. How do I request a Service Request Escalation? [...] should you still need to escalate your Service Request, simply telephone Oracle Support to update your Service Request. When you reach a Dispatcher, provide the Service Request number and ask for the Service Request to be escalated. You will be asked for a business justification for the Service Request escalation and for yours and your manager’s contact information. 3. What can you expect after you request a Service Request Escalation? If you have telephoned to request a Service Request escalation, the Service Request Escalation Owner will make every effort to call you back within 30 minutes [...] You can also at the time you call request that the SR is transferred to an engineer in your time zone. If all above fails, and you are one of my customers, call me and I will open the gates of hell on them !!!!!!! 3
Ric_ Posted June 29, 2011 Posted June 29, 2011 @linescanner I think we all know the escalation process all too well... it doesn't necessarily mean that your box gets fixed double-quick though. The speed of response really doesn't help when their answer is 'well, you're just going to have to wait until it finishes what it's doing' or 'yeah, you shouldn't turn that feature on'. As people know, I've been a long standing Sun customer and all my server estate is Sun kit... plus I've had workstations and Sun Rays. Recently though I've lost confidence in the Oracle brand. I've told them this and they don't even seem that bothered.
Butuz Posted June 29, 2011 Posted June 29, 2011 Just as an aside I've had a nightmare start to the week as one of my 7110's (serving ESXi VMs via iSCSI) has gone totally AWOL (despite busily working since 2009) and decided to stop serving up VMs to my ESXi boxes. I managed to backup all VMs and offload them onto a mix of Local ESXi storage and my other 7110. The faulty 7110 now won't even download , play or move VMs. It appears to be totally screwed. It's not covered by oracle so I guess that's the end for me. I think it has punished me for post #34. I'll be punishing it by unplugging it and removing it from my rack. Butuz
Ric_ Posted June 29, 2011 Posted June 29, 2011 @Butuz If you use the SP to view the console (or indeed plug a keyboard and monitor into it) you may find that it is just taking an age to do something and nothing. I did something and it took 2 weeks for the box to come back online!
RobFuller Posted June 29, 2011 Posted June 29, 2011 (edited) @Butuz which version of firmware was that box running out of interest if you know? Edited June 29, 2011 by RobFuller
linescanner Posted June 29, 2011 Posted June 29, 2011 @linescanner I think we all know the escalation process all too well... it doesn't necessarily mean that your box gets fixed double-quick though. The speed of response really doesn't help when their answer is 'well, you're just going to have to wait until it finishes what it's doing' or 'yeah, you shouldn't turn that feature on'. As people know, I've been a long standing Sun customer and all my server estate is Sun kit... plus I've had workstations and Sun Rays. Recently though I've lost confidence in the Oracle brand. I've told them this and they don't even seem that bothered. Agreed and the points have been raised. We are on your side and are trying our hardest to get a suitable response. I now have twice monthly calls in place to air some of you guys issues etc.
Duke Posted June 29, 2011 Posted June 29, 2011 I hate to be the odd one out here, but having worked with Andy T (Cutter), Andy P (WTL) and Phil L (Oracle) for a while on the S7000 stuff, I can say our S7410 has generally worked very well, been well supported, and been a product I'm happy I bought. I think there are a lot of happy S7000 customers out there, but generally people only post on forums when something is broken rather than when things are working perfectly. I'm sure there are a lot of S7000's out there working well, but it looks like the original S7110's and S7210's haven't been as well supported as the S7310/S7410 + S7x20 range. That's definitely not okay by any means, but it doesn't mean the whole product range is completely cr*p. My NetApp FAS2020 sucks, but NetApp's higher-end stuff is successfully used by huge organisations around the world. I hope I don't get hated on for saying this, but I still stand by the S7410 being a good product and I assume the S7x20 stuff is too. The Fishworks OS, DTrace, and the protocol licensing model are still all excellent innovations that most other storage providers are still playing catch-up to. I spoke to Phil L yesterday (sorry again for scaring you Phil) and I know he and Andy from Cutter have worked with Steve Darkes (Oracle - good guy from everything I've seen) to really try to get things moving on the issues people here are having. I just hope Oracle can turn things around because we had a great community on here around the S7000 stuff and I'm not sure we'd get the same 'personal' level of support we got from Phil Lawrence from a company like EMC or NetApp. Cheers, Chris
linescanner Posted June 29, 2011 Posted June 29, 2011 Thanks Chris. I just want to echo this, I have S7000 series customers including 7110 users that are having no issues at all. Steve Darkes (Oracle) had a long chat with me yesterday and is working hard to sort things out. Will keep you all posted
linescanner Posted June 29, 2011 Posted June 29, 2011 Have had response from Oracle. Will update in a bit. 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now