Classcharts abysmal performance today

[NetworkDirector]

I understand that this PHP vulnerability makes services need to go down, but Class Charts need to get their act together, they can't just go offline whenever they want without telling anyone, schools and customers rely on them, I guess no behavior being logged today then.

[gomalley]

2 minutes ago, Anyer said:

I never get anything as the main admin for Class Charts, yet now I know the email I did a message trace and everyone in Senior Leadership got the email bar me! When it's all up and running I'm going to get them to add me to this notification email list.

Wow! Good luck - I've asked many a time about being notified about issues and been no offering from them

[Anyer]

Just now, gomalley said:

Wow! Good luck - I've asked many a time about being notified about issues and been no offering from them

What's even more bonkers is that their notification list includes staff members that no longer work here (which I can see from the message trace), but I can't be added on?!

[Homer]

Smells like a very manual way of them having deployed their infrastructure for this length of an outage. I'm sure if they'd used up to date automation for deployment their infrastructure would have been patched and active again ages ago.

[psydii]

1 hour ago, Homer said:

Smells like a very manual way of them having deployed their infrastructure for this length of an outage. I'm sure if they'd used up to date automation for deployment their infrastructure would have been patched and active again ages ago.

I'm wondering if it is worse than that, because even pre-devops sysadmin practices ought to have allowed them to patch quicker than this.  Perhaps they have the technical debt of running on old frameworks that require production code to be re-worked before they will run on supported php versions? 

Wonde appears to be back up now. (Hadn't checked since 6am, so might even have been up by the start of the school day).

[gomalley]

3 minutes ago, psydii said:

I'm wondering if it is worse than that, because even pre-devops sysadmin practices ought to have allowed them to patch quicker than this.  Perhaps they have the technical debt of running on old frameworks that require production code to be re-worked before they will run on supported php versions? 

If this is indeed the case then thank goodness there is only 1 more day till half term - and dear god hopefully that gives them enough time

[psydii]

I have nothing to base that on, other than they are still down while others are back online.

[gomalley]

I wouldn't say years (at a guess from your post count) of experience with multiple different vendors is nothing to base it on - it is a good hypothesis - just  a shame we can't test the theory!

[Homer]

1 hour ago, psydii said:

I'm wondering if it is worse than that, because even pre-devops sysadmin practices ought to have allowed them to patch quicker than this.  Perhaps they have the technical debt of running on old frameworks that require production code to be re-worked before they will run on supported php versions? 

Wonde appears to be back up now. (Hadn't checked since 6am, so might even have been up by the start of the school day).

Ooooooh I bet you're on the money there. PHP 5.6 anyone? 😂

[3s-gtech]

They’re busy migrating mysql statements to mysqli.

[Anyer]

Had this forwarded on from one of our teachers that gets the notification emails:

 

Quote

Following our earlier email, we’re pleased to let you know that we’ve now restored a range of services, including Parents Meetings, Room Bookings and Tes Magazine. Thank you for your patience while we worked through the global issue affecting thousands of online services worldwide.

 

This was due to a global issue with widely used internet technology that sits behind thousands of online services. In order to address this, we took some services temporarily offline as a precaution.

 

Where things stand 

We’re bringing our services back online in stages, checking each one as we go. ClassCharts is expected to be available, ready for UK registration (23rd Oct). Some of our other products remain offline as a precaution while we finish our restoration work. 

 

What this means for you 

• The services listed above are available and can be used as normal.

• ClassCharts: please check back later today. When it’s available, you can use it as normal.

• Other products remain offline for now. There’s nothing you need to do, but we are continuing to work on this to resume all services as soon as possible. 

Staying safe 

We’ll never ask for your password or login details by email. If an email looks suspicious, please contact https://help.tes.com/support/home 

 
What happens next 

We’ll email you again as soon as everything is restored. 

Thank you for your continued patience. 

 

Kind regards, 
Camila Kill 
Chief Technology Officer

 

[Devontechie]

Class charts seems to be back online now

[DrCheese]

4 hours ago, Homer said:

Ooooooh I bet you're on the money there. PHP 5.6 anyone? 😂

haha yes we said this, given the complete lack of development of the system in the past few years.

[Tefters]

It's OK everybody, remember these companies are working in the best interest of schools. 🤣

 

Out of every product we buy into online this way the only one affected by using PHP and their bulk mailing supplier is so good that half the community got the emails WAY too late. I only just got the outage email at 21:30. Couldn't of simply redirected the classcharts DNS to a basic HTML holding page even if it was obvious why the outage had taken place. All this after a major wobble with the infrastructure the day before.

 

Cloud computing is a beautiful thing until it isn't.

Edited Thursday at 21:39 by Tefters

[TechMonkey]

No inside knowledge of TES, purely speculation but when you think of the number of companies they have bought up in a pretty short space of time I very much doubt system, server or code base consolidation has happened yet fully, which means potetnitally each service is run on its own instance/cluster/raspberry pi. As said, this would mean each has to be updated individually, after checking what it might affect and then run tests as well as any supporting services they may have and general websites. I can imagine that is what is taking the time, and hopefully will highlight that they need to spend some of the money they are getting from numerous price hikes on the infrastructure.

[Anyer]

Class Charts is back up, but Provision Map isn't which means none of the Pupil Passports on Class Charts are showing *sigh*

[Homer]

11 hours ago, TechMonkey said:

No inside knowledge of TES, purely speculation but when you think of the number of companies they have bought up in a pretty short space of time I very much doubt system, server or code base consolidation has happened yet fully, which means potetnitally each service is run on its own instance/cluster/raspberry pi. As said, this would mean each has to be updated individually, after checking what it might affect and then run tests as well as any supporting services they may have and general websites. I can imagine that is what is taking the time, and hopefully will highlight that they need to spend some of the money they are getting from numerous price hikes on the infrastructure.

Raspberry Pi 😆