School website down - Greenhouse School Websites - Fasthosts

[Koldov]

Anyone else got this problem?

 

Apparently there is an issue with Windows on the server hosting it...

[Brax]

Anyone else got this problem?

 

Apparently there is an issue with Windows on the server hosting it...

All good here and we use Greenhouse. Currently doing some work on it.

[Jawloms]

All fine here too.

[titch]

Yes currently the same at the school Im governor at.

[HarryMonkey]

Hi XXXX

Thank you for your patience whilst the engineers have been working on the server through the evening. I have an update for you.

 

The server came back up a few times but each time failed with a new error. We've taken the difficult decision to rebuild the Operating System from a clean image (copy). Before doing that the engineers are backing up up all of your website data to an attached drive. This is taking some time. Once complete, they will be able to wipe the drive and deploy the clean operating system image. They will then restore all the website data.

 

Unfortunately, backing up a server's worth of data, restoring the operating system and then restoring all the data will take time. I'll have a better idea of when the sites will return once I've spoken to the engineers first thing tomorrow.

 

This is the worst-case scenario we've been trying to avoid. In our 15 years of hosting school websites we've never had to do this. We spare no expense in buying top of the range servers with redundancy in case of hardware failure. Unfortunately, there is no easy switch-over for operating system failure. Thank you again for your patience and please be assured that getting these sites back online is our top priority.

 

 

Kind regards

 

. . . .

[psydii]

. ..Unfortunately, there is no easy switch-over for operating system failure.. . .

Been pondering this, and I've had a revolutionary idea, has anyone thought of separating the user data, application configuration and operating system onto their own disks? This way you might be able to swap out the OS or even the application hosting independently of user data. It could be a game changer. I should probably patent the idea.

[mavhc]

I use containers, OS takes about 10 seconds to install, and then containers will just run

[psydii]

I use containers, OS takes about 10 seconds to install, and then containers will just run

Aww shucks. Are you saying this is a long solved problem in this space?

[Koldov]

They did ask about our postcode when support were called, I wonder if they group them on the server by location (or it's a form of lookup identification on their helpdesk/support system).

 

Anyway, bit slow on the notification.... we got an email today that starts "As you will be aware, yesterday afternoon a fault appeared on the server hosting the Your School website.".... Apparently we weren't... ooops!

 

Only found out when a teacher asked me why she couldn't get on to edit the website... So I checked the filtering... no, I checked Sophos... no, I checked their website for a status page...no, so I checked the actual school website... bingo... so I checked 'Is It Down Right Now'.... and it was!

 

I'm sure we don't pay enough to get an instant failover to redundant hardware, but this is a Fasthosts issue, right? Surely someone who runs servers to host websites for a living knows how to backup/restore quickly and have configured their systems so they can...

 

EDIT: My bad, apparently "the hardware has failover options"...

Edited June 21, 2023 by Koldov

[Davit2005]

So much better ways to run these things these days and why run a web server on Windows to start with. So much more benefit and flexibility even with containers. Set persistent storage even on a separate mounted storage device/system and it is so simple to get backup and running. Edited June 21, 2023 by Davit2005

[Koldov]

Well, we've been told the website is finally up, but we're getting this...

 

 

Anybody know what the issue is?

[pete]

@Koldov

 

My guess is they've brought your site up on a server presenting a cert that's missing your domain as a SAN. Or they haven't got round to renewing the certs with Let's Encrypt (possibly because LE rate-limited them, depending on site density on the server) and because they're using HSTS (your browser knows there should be a cert because the server told it so previously) the browser's aborting the connection. If I recall correctly they mash a lot of sites into each cert.

 

--

 

As a Greenhouse customer (who's not affected) these do seem like avoidable schoolboy errors. Loading site data from backup/shared storage onto another server with spare capacity should have resolved this within a couple of hours.

Edited June 22, 2023 by pete

[Koldov]

Many thanks for confirming, that's along the lines of my initial guess too (hosting server not having correct certs) not as detailed or knowledgeable, but I was in the ball park...

 

I'm not actively involved with this (but I'll pass the info on) as it's being dealt with by the member of staff who maintains the website, however I do like to keep my ear to the ground for if it ever gets dumped on my desk!

 

I'm quite glad the receptionist phoned and said she was concerned about the message when trying to go to the website, at least some sort of cyber-security message is getting through. Although it does state in plain black and white it could be someone trying to hack you, so maybe just common sense, but even that's in short supply around here sometimes...