Network Collapsed

[papakura]

1) Well known Network Supplier to the School Sector

2) New servers bought last year.

3) Servers installed by third party

4) Network re-built by in-house technical team

5) All servers have now failed and students do not have access to their user areas.

6) GCE and some GCSE coursework deadlines have not been met.

7) Network still not up after 2 weeks.

8) NM under stress and coming into work in early hours of morning.

9) SLT not keeping staff informed.

 

What to do?

[ChrisH]

What are the specifics? People cant log on? What exactly can they do?

[FN-GM]

Like Chris says to help please can you post more details.

[GrumbleDook]

Do you have a backup of when they were installed by the 3rd party company? Presuming it was all hunky dory at that point (or at least usable!)?

[papakura]

Sorry HoD ICT so will have a limited technical knowledge.

 

The printers went down and then the servers. Problems with backup drives which were destructed and replaced then problems with RAID controllers which were replaced. But servers then failed completely so were taken away and rebuilt. Brought back in but problems persist, ie, no work stations connect to the servers. Email is down and internet is only functioning eratically.

 

The clients can be used as stand alone but printers have to be connected as stand alones etc.

 

PDFs will not open even when computers used as stand alones meaning that we cannot access digital exam papers for revision.

 

So, basically, the users cannot log on to their user area. there is a clean backup - apparently!!

[papakura]

Do you have a backup of when they were installed by the 3rd party company? Presuming it was all hunky dory at that point (or at least usable!)?

Tony,

 

I have been trying to get the third party and supplier in to fix the probs since the servers were installed last April. IMHO, there has not been enough motivation/cooperation from the triangle of vested interests (the installation team, the supplier and our tech team) to get the work done - each blaming the others bad workmanship. Personally I think they will cover each others back and the carnage wreaked for T&L will not be documented.

 

I feel sorry for our NM as he has been understaffed.

[FN-GM]

This sounds like the network has not had its upkeep as you say probably due to being understaffed.

 

What area are you in?

[pooley]

Is the server a domain controller ? Does it have the pc/users listed in active directory ?

Or has it been rebuilt and no users/pc's exist ?

 

Email down = was one of the servers an exchange server ? Has it been restored from backup ?

 

Sounds like a nightmare scenario.

 

Your NM got any help ?

 

Easy bit is the pdf's, install adobe acrobat reader or fixit pdf reader (both free) to access pdf's

[GrumbleDook]

PM on its way.

[FN-GM]

Its looks like a DC problem.

That will probably a Global Catalogue so they would explain problems if they have exchange.

It will be probably DNS so no internet connection.

They wont be able to logon

[papakura]

Thanks for the replies and it is giving me an interesting insight but why am I not getting this from the triangle of supplier/installer/Local tech team??

 

there is no way that I am going to be able to tell the NM his job, so imparting this info to him is not going to happen for that reason.

 

I guess what I am angling at is how do I

 

1) get the triangle to realise the impact on T&L and make them aware of our anger as the SLT do not seem to be doing this

2) get the Tech team and SLT to regularly update the staff/students in a useful way

3) Get advice to a similar detail that you are giving here to the ICT teachers/SLT - not happening.

4) make the SLT not expect the NM to come into work at 08:30 after leaving work at 03:00 and understand the H&S issues

5) Ultimately, get the triangle to admit the mistakes with a view to reporting on what has happened so that there is not suspicion

 

Thanks

[mortstar]

Thanks for the replies and it is giving me an interesting insight but why am I not getting this from the triangle of supplier/installer/Local tech team??

 

there is no way that I am going to be able to tell the NM his job, so imparting this info to him is not going to happen for that reason.

 

I guess what I am angling at is how do I

 

1) get the triangle to realise the impact on T&L and make them aware of our anger as the SLT do not seem to be doing this

2) get the Tech team and SLT to regularly update the staff/students in a useful way

3) Get advice to a similar detail that you are giving here to the ICT teachers/SLT - not happening.

4) make the SLT not expect the NM to come into work at 08:30 after leaving work at 03:00 and understand the H&S issues

5) Ultimately, get the triangle to admit the mistakes with a view to reporting on what has happened so that there is not suspicion

 

Thanks

 

To be fair, it does sound like, at least the NM, has an enormous amount on his plate and updating staff when so busy/stressed/overworked will not be number 1 priority.

 

As Head of ICT I'd have expected you to engage with the SLT, or the member of SLT responsible for ICT, about the terrible impact on T&L (although I expect they already know this).

 

My priorities in this situation would be:-

 

1. Getting network to useable state ASAP, disregard all else
   
2. Inform all school as to what went wrong
   
3. Leadership for ICT (Yourself, NM and ICT Co-ordinator) have enquiry into what went wrong
   
4. Learn from mistakes and ensure that something as catastrophic never happens again
   

[Grommit]

Thanks for the replies and it is giving me an interesting insight but why am I not getting this from the triangle of supplier/installer/Local tech team??

 

there is no way that I am going to be able to tell the NM his job, so imparting this info to him is not going to happen for that reason.

 

I guess what I am angling at is how do I

 

1) get the triangle to realise the impact on T&L and make them aware of our anger as the SLT do not seem to be doing this

2) get the Tech team and SLT to regularly update the staff/students in a useful way

3) Get advice to a similar detail that you are giving here to the ICT teachers/SLT - not happening.

4) make the SLT not expect the NM to come into work at 08:30 after leaving work at 03:00 and understand the H&S issues

5) Ultimately, get the triangle to admit the mistakes with a view to reporting on what has happened so that there is not suspicion

 

Thanks

 

A lot of time supplier promise you the world yet when push comes to shove do not have the technical expertise to fix the problem...

 

Your NM should be on the phone to the Suppliers and threatening to get them black listed with the LEA and other counties...

 

What is staggering is that the NM has not been able to get the network back up after 2 weeks...

 

You still do not give information of to what the status of the servers are ?

 

1/ Do the servers start ?

 

2/ Can the NM log onto the Servers

 

3/ Can you ping the servers IP from the Client PC's ?

 

4/ Can you ping the servers name from the Client PC's ?

 

5/ Do you have a redundant spare server ?

 

We all as NM's have a LOT of work on our plates but our Job is to make sure the Network is Stable and get it up again within a day or two if it falls over...

[papakura]

A lot of time supplier promise you the world yet when push comes to shove do not have the technical expertise to fix the problem...

 

Your NM should be on the phone to the Suppliers and threatening to get them black listed with the LEA and other counties...

 

What is staggering is that the NM has not been able to get the network back up after 2 weeks...

 

You still do not give information of to what the status of the servers are ?

 

1/ Do the servers start ?

 

2/ Can the NM log onto the Servers

 

3/ Can you ping the servers IP from the Client PC's ?

 

4/ Can you ping the servers name from the Client PC's ?

 

5/ Do you have a redundant spare server ?

 

We all as NM's have a LOT of work on our plates but our Job is to make sure the Network is Stable and get it up again within a day or two if it falls over...

I do not believe the servers start but will ask.

 

I will ask the NM if he can log onto the Servers

 

Can you explain the ping bit!

 

We do not seem to have a redundant spare server which is what I would have expected. Both new servers have gone and the two old ones but it does seem that they have been able to get SIMs up and running.

 

I would have thought the supplier could just bring in a new server and replicate what we have - but is that naive of me??

[GrumbleDook]

I've had a quick exchange of PMs with Papakura ... because he does not want to tread on toes to some extent as because he fully appreciates that he is not an NM some of the language we use and he uses may cause a few misunderstandings.

 

He is not sure of the exact hardware software fault, but there is an element of middleware involved. This does increase the issues with diagnosing what is wrong, and I would presume that one of the reasons why they do not have many staff is due to the middleware normally doing a chunk of the work for the school ... a number of people here take a similar approach.

 

The downside is that it is not doing the job right now and the company with the middleware are the ones taking it off-site to do stuff.

 

I think one of the key things is to try and get the NM on to EG and then those with the same middleware can help him sort things out or help him find the right levers to pull to get the middleware providers to get it sorted. If he is workign the hours it seems he is, it is not from lack of effot on his part to get it sorted.

[alan-d]

Does your NM know about this site? That may help, rather than try and muddle his way through to a cure. At the very least he'll see that he is not alone as many of us have gone through similar experiences.

 

Does your LEA offer technical assistance in the form of hiring a technician for a short period? This would help your NM. He can only work long hours for a short period before mistakes are made and he ends up taking 2 steps back to rectify a problem.

 

Keep the questions coming.

[Grommit]

I do not believe the servers start but will ask.

 

I will ask the NM if he can log onto the Servers

 

Can you explain the ping bit!

 

We do not seem to have a redundant spare server which is what I would have expected. Both new servers have gone and the two old ones but it does seem that they have been able to get SIMs up and running.

 

I would have thought the supplier could just bring in a new server and replicate what we have - but is that naive of me??

 

PIng sends a packet to a device or server with a reply to see it's available...

 

Ping with IP sees that the Network card is working.. Ping with Name sees that DNS is working..

 

Also what your supplier does and does not do depends on the SLA that you have with them...

Edited May 20, 2008 by Grommit

[papakura]

Thanks to all and especially Tony who has just helped me understand some key issues.

 

The supplier of the original server is currently on site and installing a Higher Spec server - this will cut out one side of the triangle. I presume things might be up and running tomorrow?

 

As somebody said earlier, an important next step is to explain to the staff what has happened. 4 servers going down at once seems to be improbable in the extreme?

 

I will take advice as suggested and report back to you.

 

Many thanks

[fafster]

A hardware failure in all four servers can only be explained by a power surge due to a lack of UPS. I can't see this being the case though, so it's most likely the whole domain is botched, with whatever controlling third-party software being unable to access it.

[papakura]

whole domain is botched - what does that mean in laymans terms?

[mac_shinobi]

Botched as in broken - not working - something went wrong with them.

[joe90bass]

A hardware failure in all four servers can only be explained by a power surge due to a lack of UPS. I can't see this being the case though, so it's most likely the whole domain is botched, with whatever controlling third-party software being unable to access it.

 

Have to agree here, four servers suffering hardware failure at the same time is about as unlikely as me spending the night with Kylie Minouge!

 

Have to feel very sorry for your NM! Do send him this way, if only for us to help ease his pain!

[PiqueABoo]

> problems with RAID controllers <

 

Hmm.. haven't seen very many, but some variations of RAID controller failure can be stealthy and extremely evil.

 

> whole domain is botched <

 

On your network there's a central database called "Active Directory" that holds all the information about the Windows "domain" - that's info about the user accounts, the computers, who is permitted to do what and so on and so forth. That database is critical so rather than keep just one copy on one server, it is almost certainly replicated to other servers.

 

Unfortunately it is possible for broken data to get replicated, so those multiple instances of the Active Directory database aren't much use. If that had happened I might be caught saying something like the "whole domain is [something rude]!"

[elsiegee40]

papakura says he's HoD ICT... can't help wondering if a recommendation for a website from someone in that position might not go down to well with the NM.

 

WE know it would help him, but the NM's probably thinking that a teacher's sticking his nose where it's not wanted... Hope he does come and visit. He needs some sympathy on an FFS thread!

[wesleyw]

Tony which Middleware product was he using?

 

Interesting that four servers went down simultaneously that would suggest power spike as others have suggested. 08:30 - 03:30 pretty extreme surprised the NM hadn't got it working though I should have thought get a cheap server setup AD get a list of users and get the data backup added to the server until the full domain infrastructure can be replaced and pulled back from the clean backup. A few days for an interim solution but it sound serious if two weeks has gone by! The NM must be dreading coming into work at the moment.

 

Wes

Edited May 20, 2008 by wesleyw