Problems this morning with Bromcom?

[mthomas08]

8 minutes ago, Cat_Jam148 said:

They're making SIMS look reliable.

 

If things carried on like this week after week, I would agree.

 

I am wondering if attendance is related - SIMs quite often crashed for years during early morning attendance. I still wouldn't want to repeat everything I did trying to resolve it...

[supportman]

27 minutes ago, Cat_Jam148 said:

My sympathies to all the end users who are having to put up with this on a fairly regular basis!

 

They're making SIMS look reliable.

 

Actually, I don't think SIMS was ever this unreliable...

Most of the issues I encountered with SIMS ultimately stemmed from changes I made in my younger, and less experienced, years.

 

There were _MANY_ times when Sims had similar performance issues. In fact I remember distinctly when they did a main update and crashed everyone's client due to their new homepage. and don't even get me onto the absolute mess of Solus 😂

 

Yes bromcom being down for 30mins 3 days in a row is annoying, but lets not sugar coat the years of pain with Sims

[Dr_Zeux]

4 minutes ago, supportman said:

 

There were _MANY_ times when Sims had similar performance issues. In fact I remember distinctly when they did a main update and crashed everyone's client due to their new homepage. and don't even get me onto the absolute mess of Solus 😂

 

Yes bromcom being down for 30mins 3 days in a row is annoying, but lets not sugar coat the years of pain with Sims

We've been down 5 days in a row - Tues/Wed/Thur last week and Mon/Tues this week.  Not sure there's a silver lining to this...

Edited March 10 by dbluston

[psydii]

The silver lining is that they're all that's keeping MIS's from costing £100+ per pupil per year. To be honest, what is expected of an MIS might actually cost ten times even that outside of the primary/secondary market. At some point the wheel are going to come off, either from customers fleeing, or divestment and we'll all realise the they were the good guys all along.

 

That said, I got crucified when our on prem sims behaved this badly, and I had about as much control over how it was used and what was querying it as they do. But they ought to be able to rate limit by customer / 3rd party in order to protect the overall service.

 

Ooh I've got an idea: as an option, to ensure immunity to the issues caused by others, they should offer "sovereign-cloud"  or  "on premesis" version ?  🙃

 

 

[synaesthesia]

3 minutes ago, psydii said:

Ooh I've got an idea: as an option, to ensure immunity to the issues caused by others, they should offer "sovereign-cloud"  or  "on premesis" version ?  🙃

 

 

I have that through as an idea on their portal to help protect against things like this and local internet issues - just a cache effectively which could run on any local device.

https://ideas.bromcomcloud.com/ideas/BRCM-I-7000

[Wizard101]

20 hours ago, Wizard101 said:

Yeah just how it was fundamentally built and the optimisation seems poor. Like they didn't build for any scalability or understand how much data their system would be working with and didn't plan for it. 

 

Everything requires a new window to be opened up, It feels clunky like its barely standing when I use it, you make a report it feels like it's about to die.

Dear Customer,    We want to provide you with a transparent update on the performance issues that have continued today, and the steps we are taking to resolve them.    Following this morning's peak period, our teams identified that while the infrastructure changes applied yesterday successfully increased capacity at the gateway level, some requests were then constrained by other parts of the platform further down the chain. This resulted in further errors for a significant number of customers this morning. We recognise this is frustrating, and we sincerely apologise that the disruption has continued.    We want to be clear that our teams have not stopped working on this. As part of today's investigation, we identified and addressed an additional bottleneck within the gateway layer that was contributing to the behaviour under peak load. This fix has already been applied.    In addition, our engineering teams have identified four specific software improvements that will reduce unnecessary load on the platform during peak usage periods. These changes are currently being implemented and will be rolled out progressively over the next 48 hours, subject to testing and validation.    We will continue to monitor the platform closely during tomorrow morning's register period and will provide a further update later today via Email, the Bromcom Status Page and the Community portal.    We sincerely apologise that this issue has not yet been fully resolved. This remains our absolute priority, and we are committed to restoring the reliable service you expect from Bromcom.

Edited March 10 by Wizard101

[Dr_Zeux]

13 minutes ago, synaesthesia said:

I have that through as an idea on their portal to help protect against things like this and local internet issues - just a cache effectively which could run on any local device.

https://ideas.bromcomcloud.com/ideas/BRCM-I-7000

Good job @synaesthesia!  I have already voted for this idea 🙂

[BlackCat80]

43 minutes ago, Wizard101 said:

Dear Customer,    We want to provide you with a transparent update on the performance issues that have continued today, and the steps we are taking to resolve them.    Following this morning's peak period, our teams identified that while the infrastructure changes applied yesterday successfully increased capacity at the gateway level, some requests were then constrained by other parts of the platform further down the chain. This resulted in further errors for a significant number of customers this morning. We recognise this is frustrating, and we sincerely apologise that the disruption has continued.    We want to be clear that our teams have not stopped working on this. As part of today's investigation, we identified and addressed an additional bottleneck within the gateway layer that was contributing to the behaviour under peak load. This fix has already been applied.    In addition, our engineering teams have identified four specific software improvements that will reduce unnecessary load on the platform during peak usage periods. These changes are currently being implemented and will be rolled out progressively over the next 48 hours, subject to testing and validation.    We will continue to monitor the platform closely during tomorrow morning's register period and will provide a further update later today via Email, the Bromcom Status Page and the Community portal.    We sincerely apologise that this issue has not yet been fully resolved. This remains our absolute priority, and we are committed to restoring the reliable service you expect from Bromcom.

 

Honestly, this is starting to look like outright incompetence. The same mess in September, January, and now the same mess again: repeated outages, shifting excuses, failed promises, and yet more bottlenecks only being found once schools are already being hit. That screams a technical team that does not properly understand the knock-on impact of its own changes. Whether it is weak bad change control or senior technical staff/leaders who are simply out of their depth, the outcome is the same. Bromcom looks like it is debugging their platform live in front of paying customers. For a school MIS, that is completely unacceptable and frankly embarrassing.

Edited March 10 by BlackCat80

[dmj]

52 minutes ago, BlackCat80 said:

 

Honestly, this is starting to look like outright incompetence. The same mess in September, January, and now the same mess again: repeated outages, shifting excuses, failed promises, and yet more bottlenecks only being found once schools are already being hit. That screams a technical team that does not properly understand the knock-on impact of its own changes. Whether it is weak bad change control or senior technical staff/leaders who are simply out of their depth, the outcome is the same. Bromcom looks like it is debugging their platform live in front of paying customers. For a school MIS, that is completely unacceptable and frankly embarrassing.

My reading is that it's a single system, not siloed for each school/area, so makes load testing extremely difficult with multiple single points of failure for the entire platform. 

[Theblacksheep]

1 minute ago, dmj said:

My reading is that it's a single system, not siloed for each school/area, so makes load testing extremely difficult with multiple single points of failure for the entire platform. 

 

Split the UK up so tat least it is isloated, other areas have rather a greater uptime or no one is noticing so bromcom dont update their gaslighting chart.

[dmj]

18 minutes ago, Theblacksheep said:

 

Split the UK up so tat least it is isloated, other areas have rather a greater uptime or no one is noticing so bromcom dont update their gaslighting chart.

It goes back to @psydii point that they are doing this on the cheap. 

[BlackCat80]

100%... sure... 🤣

[pete]

Bromcom have discovered in-browser caching and apparently would love it if you gave them a cookie for their efforts:

 

Our engineering teams have implemented changes designed to reduce the load on our platform during peak usage periods. Specifically, we have ensured that static files, such as icons and interface elements, are now cached locally within the browser. This reduces the number of requests made to the server and prevents the platform from working harder than necessary during busy periods. 

 

I have Y7 CS students who would be ashamed of having to make this announcement.

[BlackCat80]

10 minutes ago, pete said:

Bromcom have discovered in-browser caching and apparently would love it if you gave them a cookie for their efforts:

 

Our engineering teams have implemented changes designed to reduce the load on our platform during peak usage periods. Specifically, we have ensured that static files, such as icons and interface elements, are now cached locally within the browser. This reduces the number of requests made to the server and prevents the platform from working harder than necessary during busy periods. 

 

I have Y7 CS students who would be ashamed of having to make this announcement.

That honestly makes it sound worse, not better. Browser caching for static assets is basic engineering and I am pretty sure it was already there long before last week. I even remember this being looked at with Smoothwall who were investigating our firewall issues a few years back. Surely something changed last week that caused this. These assets are not new. Either Bromcom broke or bypassed caching with a recent change, feels like they are dressing up as if it is a new fix. Neither reflects well on their technical teams. Very bad!

[dmj]

[Synkrox]

Yeah I recall browser caching issues, but funnily enough not recently... I'm sure there have been a couple of times where "delete cache files" has been a fix for things. But I actually can't remember when that was last the case.🤔

 

Be nice to relive the old days again if they are bringing it back! 🫠

[pete]

 

(or women, or thems, or haven't-decided's.  Incompetence isn't gender-specific)

[Marci]

JS and CSS assets have been failing to minify server side and been delivered in full-fat form for at least 2 years or so. At a glance there were only ever a handful of actual image assets. Rest of the payload is either from XHR calls or fonts.

Edited March 11 by Marci

[synaesthesia]

Has anyone else's Bromcom stopped certain bits on the MCAS app working? For example we've had a few parents call in to say the attendance dashboard no longer shows anything for their kids, and have verified that with my own too

[Synkrox]

4 minutes ago, synaesthesia said:

Has anyone else's Bromcom stopped certain bits on the MCAS app working? For example we've had a few parents call in to say the attendance dashboard no longer shows anything for their kids, and have verified that with my own too

Yeah I popped it on the BC community yesterday and logged a ticket

 

Edit: Looks like it might be working again? Still giving some 404 errors on dev console but otherwise pages seem to be loading. Couldnt get into messages/inbox yesterday

[Marci]

By the looks of things they're subdomaining bits out... eg:

• CloudMis odata now gets syphoned off to misodata.* (this was put in place during the last round of 429s back in Nov I think)
• Bits of CloudMIS itself randomly get redirected to appsapi.* (this has started happening since the current mitigation was put into effect)

...gives them finer grained traffic shaping / resource provisioning etc rather than the one overarching set of rules / restrictions applied at cloudmis.* level applying to everything passing thru that front end request router, as well as redistributing the load in general.

 

I suspect the knock on effect of this is impacting endpoints used by the mobile apps etc

Same is happening within bits of CloudMIS eg: builtin reports all fail if you happen to end up on the appsapi.* URL via signpost-switching across schools as can't CORS across to the reporting engine at mis-reporting.*  (workaround: logout, log back in, go direct to school in question. Pretend the signpost-switcher doesn't exist)

Minification still fails on majority of generated css assetts and bloats them up to be larger than if minification was never attempted due to the debug output being written into the final delivered files (always has).

Edited March 12 by Marci

[supportman]

I was also pretty shocked at that announcement. Makes them sound really amateur when image caching issues are blamed. I mean that's basic website stuff.

 

The whole system needs to be split up and streamlined.

 

Their product is amazing, but its clearly not scaling well.

[BlackCat80]

3 hours ago, Marci said:

By the looks of things they're subdomaining bits out... eg:

• CloudMis odata now gets syphoned off to misodata.* (this was put in place during the last round of 429s back in Nov I think)
• Bits of CloudMIS itself randomly get redirected to appsapi.* (this has started happening since the current mitigation was put into effect)

...gives them finer grained traffic shaping / resource provisioning etc rather than the one overarching set of rules / restrictions applied at cloudmis.* level applying to everything passing thru that front end request router, as well as redistributing the load in general.

 

I suspect the knock on effect of this is impacting endpoints used by the mobile apps etc

Same is happening within bits of CloudMIS eg: builtin reports all fail if you happen to end up on the appsapi.* URL via signpost-switching across schools as can't CORS across to the reporting engine at mis-reporting.*  (workaround: logout, log back in, go direct to school in question. Pretend the signpost-switcher doesn't exist)

Minification still fails on majority of generated css assetts and bloats them up to be larger than if minification was never attempted due to the debug output being written into the final delivered files (always has).

That sounds entirely believable, and if true it actually makes Bromcom’s position look worse, not better. It suggests they are pushing bits of traffic onto different subdomains as a live mitigation, then discovering knock-on issues with CORS, reporting, mobile endpoints and signpost switching as they go. That is not a platform under control, that is a platform being patched in production. Add in the long-standing CSS/minification mess and it points to the same conclusion again: this is not just a capacity issue, it is deeper problems.

Edited March 12 by BlackCat80

[Synkrox]

1 hour ago, BlackCat80 said:

That sounds entirely believable, and if true it actually makes Bromcom’s position look worse, not better. It suggests they are pushing bits of traffic onto different subdomains as a live mitigation, then discovering knock-on issues with CORS, reporting, mobile endpoints and signpost switching as they go. That is not a platform under control, that is a platform being patched in production. Add in the long-standing CSS/minification mess and it points to the same conclusion again: this is not just a capacity issue, it is deeper problems.

 

It's a similar to the journey I go on when I'm vibecoding docker compose YAML with Claude.

 

(for the avoidance of doubt, this is not a compliment)

 

[BlackCat80]

Quote

11/03/2026 15:45

Dear Customer,  

As mentioned in our last correspondence on 10th March, we are providing regular updates until our performance issues are fully resolved. Here is the latest update following work completed last night.  

What we have actioned  

Our engineering teams have implemented changes designed to reduce the load on our platform during peak usage periods. Specifically, we have ensured that static files, such as icons and interface elements, are now cached locally within the browser. This reduces the number of requests made to the server and prevents the platform from working harder than necessary during busy periods.  

We have gradually rolled this out to all customers while monitoring performance closely at each stage. Since applying these changes, we have successfully achieved a reduction in server load, and morning registration was seen to be flowing as normal. Based on the improvements observed, we are confident that the issues experienced over the last week have now been mitigated.  

Next steps  

We will continue to monitor platform performance closely over the coming days. Alongside this, we are implementing further key improvements including software optimisations and infrastructure enhancements to ensure the platform remains stable, resilient and performant as usage continues to grow.  

A full Root Clause Analysis will be shared via email within the next 14 days to all registered support users and also via the Bromcom Status Page and the Community Portal.  

We apologise for the disruption and inconvenience this has caused. We understand how critical reliable systems are to school operations, especially safeguarding. Protecting children and supporting the schools who care for them is a responsibility we take extremely seriously, and we remain committed to strengthening the reliability and support you depend on.  

Kind regards,  

 

Osman Erener    

Chief Operating Officer  

Bromcom Computers Plc  

 

14 days for a Root "Clause" Analysis?