Problems this morning with Bromcom?

[Jaan]

Bromcom gets a F for uptime;

 

https://schoolsweek.co.uk/mis-outage-leaves-schools-in-the-lurch/

[supportman]

So apparently the problem was photos for students in the registration app!

 

I've heard that somewhere before (Looking at you Sims!!)

 

EDIT: Frustrating to see Bromcom trip up on something so obvious.

Edited September 12, 2025 by supportman

[Synkrox]

17 hours ago, mavhc said:

So the architecture of their brand new cloud first system is: run a load of windows servers in a domain and an mssql database for each school?

 

I honestly think it's more like:

 

"So the architecture of their brand new cloud first system is: run a windows server and an mssql database?"

 

It's the only explanation for everyone experiencing exactly the same capacity issues.

[Cat_Jam148]

1 hour ago, supportman said:

So apparently the problem was photos for students in the registration app!

 

I've heard that somewhere before (Looking at you Sims!!)

 

(Please don't sue me)

[TheCookieMonster]

20 minutes ago, Synkrox said:

 

I honestly think it's more like:

 

"So the architecture of their brand new cloud first system is: run a windows server and an mssql database?"

 

It's the only explanation for everyone experiencing exactly the same capacity issues.

 

Bromcom's been Cloud-based for quite a long time. Far from "brand new". VMs with SQL Server per tenant is a bit more lift-and-shift though. It's not really SaaS architecture (e.g. kubernetes + serverless everything). It's more like a datacentre in the cloud but is that inherently bad? 🤷‍♂️ you can put bad, inefficient code in a "true SaaS" system too. 

 

 

[psydii]

To me, its not so much the architecture of the systems, its that they don't appear to have tested against predictable workloads and then act all surprised when the service buckles.

 

At the end of the day, whether its one sql server or hundreds of containerised microservices, ultimately its all running on shared cpu/memory/disk. If some common transaction, or underling piece of hardware that every school's instance uses becomes non-preformant, everybody feels it.

[Jaan]

11 minutes ago, Cat_Jam148 said:

 

(Please don't sue me)

 

Lol awesome

[Alis_Klar]

Interim Root Cause Analysis Performance Issues (12/09/2025) has been published.

 

[machy]

Better than I expected from them, but still meh 

 

More companies need to learn from Cloudflare and there incident reports, should really be the minimum standard

[synaesthesia]

Quite a decent report, the level of transparency I like to see. 

Remember boys and girls, it's OK to make mistakes, and it's very OK to put your hands up, admit they were made and learn from those mistakes.

And that last bit is very important!  Fingers crossed it stays that way!

[TheCookieMonster]

The RCA is quite solid and it is interim. Expect we will get more detail later along with more remedial action. 

Seen some slowness of MCAS has been reported on the status page this morning, but I'm not seeing any of that.

Good to see them highlighting stuff to warn anyway. Probably quite on edge!

  

 

[Alis_Klar]

"code-level optimisations" Badly needed.  When you click student or Staff in Bromcom (the most used parts) the system ALWAYS loads the full table of ALL students or staff with potentially 10 or so fields per student.  BEFORE it lets you even search a student.  This must add a lot of unnecessary load to the the system.

 

It's not just an infrastructure issue.

Edited September 15, 2025 by Alis_Klar

[Dr_Zeux]

1 hour ago, synaesthesia said:

Quite a decent report, the level of transparency I like to see. 

Remember boys and girls, it's OK to make mistakes, and it's very OK to put your hands up, admit they were made and learn from those mistakes.

And that last bit is very important!  Fingers crossed it stays that way!

Meanwhile, there's performance issues again this morning...  Anyone else having issues??

[Alis_Klar]

4 minutes ago, dbluston said:

Meanwhile, there's performance issues again this morning...  Anyone else having issues??

Yeah, slowness reported for registers across our schools.  Back to normal now.

[Dr_Zeux]

1 minute ago, Alis_Klar said:

Yeah, slowness reported for registers across our schools.  Back to normal now.

Good, not just our school then.  I agree, seems to be back to normal again now...

[supportman]

Glad to see they finally separated out the API access for syncs. That exponentially causes issues at scale and should never take down the main system.

 

If there is a capacity issue then the API should fail, then the client app, then the server app.

 

And agree about the code optimizations needed, loading full tables of students is the very basic stuff you shouldn't do.

 

Also they need to look at registration data flow. Thumbnails are always an issue but also the traffic it creates on client and server side.

[psydii]

If it took four days to do all that, I wonder why they didn't spent any four days  *before* the start of term  doing that?

 

Start of term work-loads are 100% predictable. This was avoidable. (Mostly. Instances failing under load due to flawed implementation is testable/avoidable, failures in the underlying cloud providers infra-management (recycling azure components), less so - but testing their own code properly would have enabled them to avoid entering that failure state/path in the first place.

[DrCheese]

Will have to forgive me as I don't totally understand it - But why isn't each school/setup in an entirely seperate container/Docker setup? 

Best I could tell the other day they had a single server crash out that took down everything, which indicates they're treating servers like Kings (Uncrashable/super important) instead of cattle (Easy to replace/spin up/unimportant)

[Rob_D]

14 minutes ago, DrCheese said:

Best I could tell the other day they had a single server crash out that took down everything

Could be a host went down taking a bunch of instances with it.

[supportman]

20 hours ago, DrCheese said:

Will have to forgive me as I don't totally understand it - But why isn't each school/setup in an entirely seperate container/Docker setup? 

Best I could tell the other day they had a single server crash out that took down everything, which indicates they're treating servers like Kings (Uncrashable/super important) instead of cattle (Easy to replace/spin up/unimportant)

 

Yes I believe its a monolith architecture.

 

There is nothing actually wrong with that, you just need to design the devops stuff to have no single point of failure.

[sefton]

Got to say, every time I have a small issue with Arbor I come back to the start of the academic year and how Bromcom have constantly had issues at this point and I'm convinced I made the correct choice at procurement.

[pete]

The full RCA has just been sent out to the school/trust main contact.

[machy]

Can be seen here

Final Root Cause Analysis Sept 2025

 

The TL;DR is, they didn't scale up enough and didn't do enough load testing, mainly caused by crap code. 

[Synkrox]

Kudos to them for keeping it real and not suggesting they will change anything to stop it happening again, or to improve performance in the future.

 

[Cat_Jam148]

I look forward to reading a repeat of this all again next year. I feel we missed a huge bullet going with another MIS.

5 minutes ago, Synkrox said:

Kudos to them for keeping it real and not suggesting they will change anything to stop it happening again, or to improve performance in the future.

 

They're too busy chasing more schools. There’s probably an engineer who’s been shouting ‘this is a problem!’ for years, while management responded with the time-honoured tradition of ignoring it until it became someone else’s emergency.