Network crash

[sluggster66]

Hi

This is my first post so will try my best to explain myself.

 

We have recently had our servers vertulised with Microsoft hyper v, we also run these in a failover cluster. This work was done by an outside firm, who were really good.( we have someone come in for 3 hours a week to help with the network manager bit, otherwise it's just me, and I have very limited experience of the server side of things)

 

Over the weekend our network has gone down, the head of I.T was in and said he couldn't see any of our VMs, and all the nodes are saying they are at critical. You can't remote into any of the vms. And even logging into a physical DC took along time. I should add that our main DC and FSMO roles are all on a VM.

 

He has managed to see a couple of VMs after various reboots and these are saying saved-critical. Still cannot access these. Failover cluster manager saying not ready.

 

Any ideas as to what I can do once I am in on Monday morning would be great.

 

Thanks in advance

Simon

[plexer]

Yeah get the company who built them in.

 

Ben

[sluggster66]

We will, but that won't be before staff need access to the network. So if there is anything that can be done in the meantime??

[bossman]

Yes there is!

 

Ring up the company who did the install as it seems they didn't do such a great job after all!

 

I take it you have not paid them yet and you did get a period of time before you signed the job off?

 

Is the head of IT a technical person or a teacher?

 

If its a teacher then I wouldn't let them anywhere near as they could do more damage, get the company who did the install to sort it all out under the terms of the warranty of install.

 

Good luck

[Roberto]

We will, but that won't be before staff need access to the network. So if there is anything that can be done in the meantime??

 

Well there probably is, but you've not really described the problem to us in any useful manner so no one here is in a position to advise you in detail.

The question is WHAT failed

• Is there shared storage that might now be offline? For that matter, whatever kind of storage is in use, what state is it in?
  
• Could there have been a powercut and a few inter-dependant services didn't start up right?
  
• Could a switch be down? Cable damaged?
  
• Have the hosts been restarted?
  
• What does the 'head of IT' mean when they say they can 'see' the VMs? 'See' them from where? Are the VHD files, etc. available for these virtual guests?
  

Edited September 16, 2012 by Roberto

[sluggster66]

Well there probably is, but you've not really described the problem to us in any useful manner so no one here is in a position to advise you.

The question is WHAT failed

[*]Is there shared storage that might now be offline?

[*]Could there have been a powercut and a few inter-dependant services didn't start up right?

[*]Could a switch be down?

 

Fair point, I suppose the answer is I don't no. This has been relayed to me by phone briefly this evening. As far as I know it's not a switch problem and he didnt think there had been a power outage. Being new to I.t support this is well out of my comfort zone and I was really just looking for ideas from people. If that isn't possible because of the amount of info I have given fair enough.

[plexer]

Even if members here gave you advice based on what information you have provided or could provide it's unlikely that you'll be able to do anything until the "experts" are on site anyway.

 

What level/response do they offer for failures with their setup as ideally they should have been in today to begin sorting it.

 

Ben

[Roberto]

 

Fair point, I suppose the answer is I don't no. This has been relayed to me by phone briefly this evening. As far as I know it's not a switch problem and he didnt think there had been a power outage. Being new to I.t support this is well out of my comfort zone and I was really just looking for ideas from people. If that isn't possible because of the amount of info I have given fair enough.

 

Well the bulletpoints I listed are a reasonable place to start troubleshooting.

 

Nothing wrong with being new to IT and/or being out of your comfort zone, it happens to us all. This problem happening now while the system is new will hopefully mean the installers will fix it for you without too much trouble (in fact I'd be reluctant to do too much yourselves for precisely this reason), but it should also serve as a warning shot to your school that they need to get you up to speed on this technology so you can support it properly yourselves, to some degree at least.

[bossman]

@sluggster66:

 

I wouldn't touch anything at all prior to the staff coming in as this could void your warranty with the company who have done the install and hence why I offered the advice I did,

 

Its alright for others to give what seems as good advice in terms of what to do to pinpoint the problem but they are not you and they are not having to suffer the consequences should any of their advice goes wrong for you.

 

Advice is said to be free but not in your case as it could cost your school and you severely, should it all go badly, the company can and will shift the blame if they can.

 

Not having a pop at anyone on here but this is what we all would do in @sluggster66 place, I know I would

[sluggster66]

I appreciate the advice, I think I will leave it to the experts to sort.

 

Thanks for taking the time to reply.

[plexer]

If there isn't allready ask them for documentation and for a full report on what happened.

 

Ben

[witch]

@sluggster66

My advice? DON'T let the panicking teachers and assorted staff spook you into mucking about or trying ANYTHING. Just quietly say that you have called the experts and they will be in to sort it.

They can't have access to the network until it is sorted. End of story I'm afraid.

 

*and welcome to edugeek - you will find us friendly and helpful, and quite a few of us couldnt survive in our jobs without the clever guys and girls on here!

Edited September 16, 2012 by witch

[twin--turbo]

We have had a few failures on our Xen Clusters.

 

We now know our way round them, and it helps the installer when we can tell them what happened and what we have checekd.

 

Then we leave it to them to fix as we pay them for that.

 

And if it takes half a day then the staff have reverted to their "Alternate Lesson Plan" and if anyone get's annoyed then the SLT back us up.

 

Although we are capable of fixing our system, we pay to make sure we are not liable for it.

 

Rob

[speckytecky]

@witch is spot on @slugster66 sounds like it's completely the responsibility of the folk who did the installation. As a matter of interest where in the UK are you and how big is your school?

 

@sluggster66

My advice? DON'T let the panicking teachers and assorted staff spook you into mucking about or trying ANYTHING. Just quietly say that you have called the experts and they will be in to sort it.

They can't have access to the network until it is sorted. End of story I'm afraid.

 

*and welcome to edugeek - you will find us friendly and helpful, and quite a few of us couldnt survive in our jobs without the clever guys and girls on here!

[sluggster66]

Hi

just to update the situation. We currently have the company, that did the work, in looking at the issue.

 

 

Thanks again for the replies

[cpjitservices]

Let us know what the issue was/is.

[sluggster66]

Hi

Just to update, the issue turned out to be a switch failure.

The network was up and down for about a day and a half, although most of the disruption was on the Monday morning. The company that did our migration replaced the switch and everything looks ok.

 

Thanks again for the responses

Simon