Network crash - Smoothwall, Switches or Server! Help!

[TMBS]

Hi everyone

 

Our school network crashed this afternoon, users not able to log on, PCs freezing, no network or Internet access. It was mostly whole site but some computer were running okay.

 

I checked the switches (Netgear unmanaged), connections seemed okay though one switch was flashing green lights rapidly. I rebooted the switches and Smoothwall box but the problem stayed the same. Rebooted the two servers, server one took 15 mins to shut down, server 2 much longer. Once up and running network access seemed to be restored but still no Internet whole site.

 

I then spent two hours on the phone to Smoothwall, they couldn't connect to the Smoothwall box so were getting me to manually reset all the ports. We couldn't even connect a laptop to the box and connect to it and then I got chucked out by Premises.

 

I now have to go back to work knowing it's not much better tomorrow. What else can I try? Is there anyway of checking our switches? Or the router? If I unplug Smoothwall and connect a laptop to the router will I be able to rule out the router?

 

I'm not sure what to do next and I have very limited knowledge of networking...

 

Any help will be most appreciated!

 

Thanks

Tracey

[Liam]

Is dhcp working and giving out the right gateway address? Have you got any possible loops in the network? Is DNS working correctly? Is it that PCs aren't getting the correct policies from group policy? Maybe try assigning a static IP and manual proxy and trying that way? Have you contacted your isp to confirm connection status?

[HPlum78]

hmmm so much going on to be honest some clients still working but servers and other clients not. Where to start! so you have smothwall that is a filter/ web proxy if my memory serves and could just be unplugged to be fair and eliminated first thing. Could do with knowing bit more detail around servers like OS ver HW and the likes, what is the core network infra makes and models and the like just the broad strokes no need for the finite detail.

 

oh and client os

Edited September 15, 2016 by HPlum78

[TMBS]

Hi Liam

 

I can ping the server and internal pcs but not external addresses, or Smoothwall. When I run ipconfig /all it shows the correct ip address and gateway on my PC so I presume that DHCP is working correctly? Is there any other way of checking?

 

We didn't have any loops in the network and haven't had any work done today but who know what students will do...

 

I've not run a nslookup, will do tomorrow. Is there any other checks I can run on DNS?

 

I rang KCOM our ISP and they confirmed there are not issues at present.

 

Thanks

Tracey

[HPlum78]

Sorry just trying to get the whole picture here, when you say clients have no network has that stopped all connectivity or is it just they cannot get to the internet? when there server(s) took so long to reboot after did your client connectivity improve? Edited September 15, 2016 by HPlum78
typing with my face again!

[TMBS]

We have two servers (HP Proliant) both running 2012 and we have a RM network, most of our PCs run Windows 7 but a few have 8.

 

We have the fiber, KCOM router, and main netgear (sorry don't have models to hand) in our office, then five ICT rooms all with Netgear switches that are cabled back to the main switch in our office.

 

Thanks

[Liam]

If you can't ping the smootwall device which I assume is your gateway address it suggests that's possibly your issue? Has the appliance lost its config potentially? Has it been updated recently? Can you plug a laptop into it with a static ip and communicate with it that way? Have you confirmed your smootheall appliance has defo got network connectivity ? Does this run from the core switch? I know you said unmanaged but you defo don't have any vlans?

[TMBS]

When the network first crashed clients had no network/Internet connection. After rebooting the servers the network connection was restored to the PCs I checked (didn't have time to check whole site) but no Internet connection.

[Liam]

Can you ping your DNS server? I assume you have this roll on one of the two servers if not both?

[Liam]

Can you get out to the Internet on the servers. Sounds very much like there is an issue with the smooth wall appliance to me. If you can't ping that then there is networking problems between that and your network. Often us Ict folk try and thing of the most complicated thing it could be. This is where the osi model comes in. Just try not to overthink it and you will sort it.

[HPlum78]

My thinking is down the same lines as @Liam around the config of the smothwall HW or config. And like he says first place to start is direct connection out to the inter web from the router.

[Liam]

Agreed ... so first thing go to your server room / office where it is located and checks for physical connectivity. If you have that, plug your laptop directly in to the smootheall device with a static IP in the same range. If you get a reply on a ping then try and get out to the Internet, ensuring your gateway address is correct.

If you establish connection to the Internet directly connected you know you can rule that out. If that's the case, jump back on here and I'm sure we can all sort it with you! [emoji6]

[HPlum78]

Not being able to connect to the smoothwall device from a laptop plugged in while on the phone to their tech support! I guess that may be the root cause just there......... I am guessing they got you to reboot that box? what was their tech supports thoughts around why you could not connect?

[TMBS]

Yes I can ping my DNS server and remote onto it, so I'm thinking it's not a DNS, or DHCP issue. If I unplug our Smoothwall box from the network and just connect the router straight to the network will that work? Not sure what setting were changed when it was installed...

 

Thanks both for your help. I feel a steep learning curve coming on and I won't be able to Google anything once I get into school!

[TMBS]

When the servers were back up I could connect to Smoothwall for about 10 minutes and then it went. They also can't access it from there end either, so I'm having to log into the Smoothwall box itself and follow their instructions. We've rebooted, copied files and reset all the ports. Once reset they got me to connect via a laptop with static IP but it still didn't work...

 

I thinking of a career change

[HPlum78]

The only issue with that is the potential of direct internet access to all of your clients that in your 11-16 school may not be the best plan. We in the Uni's don't have the same issues and are not bound by the same rules! With what you have just posted it looking more like the SW box is the issue here. Edited September 15, 2016 by HPlum78

[HPlum78]

You could cobble it together and like you suggest bypass the SW box, this would be a major risk in a school setting one I don't think I would be prepared to carry. I would guess this would have to be down to your heads appetite for internet access balanced against the risks of implementing that setup! only you guys can answer those sort of questions.

[johnat1121]

I Wouldn't discount a network loop still as this would explain the flashing of the switches initially and the server taking a long time shutting down trying slowly to close all active connections whenever we have a loop on our smaller site the smoothwall and router will give up after a few minutes and not return even once the loop has gone it may be worth turning off the core switch leaving just the servers and smoothwall connected, rebooting the smoothwall then ping -t it while connecting each edge switch back up and if the pings stop you have the cause, on an unman-aged network its not just a physical loop it could be a dogey network card or switch e.c.t.

[Liam]

You will have to let us know the outcome

[Liam]

Yes I seen that before with a dodgy nic on a really old web server we had years ago! Causes all sorts of issues.

Also ensure there are no other devices handing out dhcp addresses.

A lot to take in there but it gives you an idea.

Might be worth reviewing the network setup to prevent things like this. If you are unsure do you have a next level support company that could help?

[mrbios]

Netgear unmanaged

 

Two of my most hated words in networking, and you've combined them! If it's not the smoothwall (i assume it's acting as your internal router in this case?) A loop wouldn't surprise me.

[pantscat]

Can you plug a keyboard, mouse and monitor into your smoothwall box? Not sure if the appliances have this as I've always run it on HP tin or VM.

 

If so - then you'll be able to see if it can ping other devices - it may be that it's got upset and lost some of its config.

[hallb15]

I Wouldn't discount a network loop still as this would explain the flashing of the switches initially and the server taking a long time shutting down trying slowly to close all active connections whenever we have a loop on our smaller site the smoothwall and router will give up after a few minutes and not return even once the loop has gone it may be worth turning off the core switch leaving just the servers and smoothwall connected, rebooting the smoothwall then ping -t it while connecting each edge switch back up and if the pings stop you have the cause, on an unman-aged network its not just a physical loop it could be a dogey network card or switch e.c.t.

I agree - a loop seems the most likely cause if nothing else has changed. There's been a lot of storms around recently so it could be a dodgy network card or more likely someone has plugged both ends of a network cable into the network. Which switch was flashing constantly? This is most likely where the problem lies.

As @johnat1121 suggests, turning all the switches off and then step by step turning them back on, starting with the core switch (the one that your servers are connected to), should help you trace the fault.

Please let us know how you get on.

[Homer]

Is the Smoothwall the primary router for your LAN, are you vlan'd out or flat network? What is handling your DHCP, Smoothwall or server?

 

Are you getting timeouts or unreachables if you ping, is DHCP working properly and can you look up internal DNS addresses?

 

If you can't reach the Smoothwall internal IP and you've not changed anything I would suspect something potentially hardware level has gone wrong with it. Do you have config backups?

[3s-gtech]

Agreed with others here - this still sound like a loop. Easy for students to do without anyone noticing, especially with unmanaged switches.

 

Best way to check with this is to unplug cables in your central comms cab/server room to each edge switch, wait a little while, then see if anything comes back. Start with the switch that's flashing rapidly.

 

I've had loops take down the entire network in years gone by, and also had a particular PC with dodgy NIC broadcast storm the whole site offline. Not fun. Not that hard to trace though.