Problems this morning with Bromcom?

[STK91]

6 minutes ago, Bromcom_Alec said:

The system is now fully restored, but if you are still experiencing issues please contact Support.

Until tomorrow when it will be something else. Not noticed you seem to have a lot of issues with Bromcom? 

[simpsonj]

Was working beautifully, until about 2 minutes ago...

[machy]

Gone again for us too...

> SupprisedPikachu.gif 

[Bromcom_Alec]

Thanks for raising this. Our major incident team is on the case.

[Olliedawg]

Wooohooo

[synaesthesia]

I wonder how carefully calculated the SLA's are to avoid having to actually cough up responsibility for these failures... and if the SLA's should be tightened around key parts of the school year.

[Alis_Klar]

Up and down like a yo-yo this morning

[k-strider]

same here i want to run a report but cant

[k-strider]

[Theblacksheep]

Bromcom is guarenteed to be down when you need it.

 

If only user spikes and load could be predicted.

[Rob_D]

and ours has gone splat again. 😓

[Dr_Zeux]

Yes, our 4 x schools are also down this morning, for a second day in a row...

[machy]

9 minutes ago, synaesthesia said:

I wonder how carefully calculated the SLA's are to avoid having to actually cough up responsibility for these failures... and if the SLA's should be tightened around key parts of the school year.

My guess is clock hours instead of Working hours and  the outage clock starts ticking when they admit theres an issue 

 

 

8/168hrs rather than 8/40 

[Alis_Klar]

5 minutes ago, Theblacksheep said:

Bromcom is guarenteed to be down when you need it.

 

If only user spikes and load could be predicted.

Thing is they tried to anticipate this but one of their servers was bad and it still took down the lot.  They admitted as much on their status page yesterday but it has been scrubbed today.

[Dr_Zeux]

On 04/09/2025 at 09:58, E_G_R2 said:

Every damn year it is just not good enough for such an essential service, safeguarding and all. 

4th year in a row I believe...

[Alis_Klar]

• Update to this Incident

  posted on Sep 05 2025, 10:12 BST

   

  The system we be restarted at 1030AM today (05/09/2025) in order to restore normal service. This will mean users will lose access to the system for a short period. We apologise for the inconvenience this causes and the ongoing issues today.

   

  Why does this need to be "re-started" like its a single SQL service for the whole edifice!!  Does Normal service means yo-yo response times?

Edited September 5, 2025 by Alis_Klar

[dmj]

I'm sure they'll send out a message at 4pm saying "all systems normal" like they do every year for the first week in september

[STK91]

19 hours ago, STK91 said:

Until tomorrow when it will be something else. Not noticed you seem to have a lot of issues with Bromcom? 

Ah, how did I know....

[STK91]

22 minutes ago, Bromcom_Alec said:

Thanks for raising this. Our major incident team is on the case.

It's just not good enough. Fix the on-going issue!!

[TheCookieMonster]

3 minutes ago, Alis_Klar said:

Thing is they tried to anticipate this but one of their servers was bad and it still took down the lot.  They admitted as much on their status page yesterday but it has been scrubbed today.

It's not been scrubbed - still there from 1143 yesterday - https://status.bromcomcloud.com/incidents/7fca863c-9444-4041-8b9f-8588423f5c93

 

"On Sunday 31st August, we increased the resources on our cloud system to help make the return to school run as smoothly as possible. Unfortunately, on Thursday 4th September, one of our servers developed a serious fault. This has caused some ongoing instability when users are connected to that server. At the moment, we aren't able to take it out of action ourselves, so we've raised it as an urgent incident with our infrastructure supplier to get it fixed and restore stability for everyone."

 

So it's an infra problem, not capacity. Impact is the same however unfortunately. The nuance will not be important to most.  

 

[Dr_Zeux]

5 minutes ago, Alis_Klar said:

Thing is they tried to anticipate this but one of their servers was bad and it still took down the lot.  They admitted as much on their status page yesterday but it has been scrubbed today.

Wow - you're kidding!?  That's a bit scandalous, as the reason they're giving to customers didn't sound like that was the issue!

[dmj]

1 minute ago, TheCookieMonster said:

 

"On Sunday 31st August, we increased the resources on our cloud system to help make the return to school run as smoothly as possible. Unfortunately, on Thursday 4th September, one of our servers developed a serious fault. This has caused some ongoing instability when users are connected to that server. At the moment, we aren't able to take it out of action ourselves, so we've raised it as an urgent incident with our infrastructure supplier to get it fixed and restore stability for everyone."

 

That's a bad architecture. We suffer server outages and it NEVER affects the applications, let alone taking out the entire service. Even a primary DB shutdown shouldn't do this, the replicas just autopromote.  

 

[Alis_Klar]

6 minutes ago, dbluston said:

Wow - you're kidding!?  That's a bit scandalous, as the reason they're giving to customers didn't sound like that was the issue!

Sorry -- @TheCookieMonster has corrected me, it wasn't scrubbed.  Seems I can't use the status page. https://status.bromcomcloud.com/incidents/7fca863c-9444-4041-8b9f-8588423f5c93

 

@TheCookieMonster

Edited September 5, 2025 by Alis_Klar

[k-strider]

[Dr_Zeux]

5 minutes ago, dmj said:

That's a bad architecture. We suffer server outages and it NEVER affects the applications, let alone taking out the entire service. Even a primary DB shutdown shouldn't do this, the replicas just autopromote.  

 

I totally agree - this is the point I was making to my Helpdesk ticket yesterday!  How can 1 server developing a fault cause this kind of impact??  Surely, there are multiple servers that simply take over, and balance the load.  Why are such a big organisation like Bromcom, who host thousands of schools, so reliant on 1 server???