[bromcom] Bromcom Down - 20/5/24

[TheRobins]

Morning, Anyone else un-able to get into Bromcom this morning?

[CHiLL]

Yeah, we're unable to load it at all here. Can't even log in to see what server we're connected to.

 

Update: They've just posted about it on their community forum: https://community.bromcomcloud.com/bromcom-announcements/post/known-issue---bromcom-mis-speed-20-05-2024-RmNzm85hTpT61lO

[TheRobins]

Last place I think of looking, just had a OneUptime post and it says resolved, seems to be up but very slow now. 15 minutes to get right before registration and paper registers

[Rob_D]

Last place I think of looking, just had a OneUptime post and it says resolved, seems to be up but very slow now. 15 minutes to get right before registration and paper registers

To be fair, I just come straight here.

 

The support/status page is still saying they're investigating the issue: https://status.bromcomcloud.com/

[Bromcom_Darren]

Good morning all,

 

As stated on the Community and the status page the team are aware of this issue and are investigating as a matter of urgency.

 

Darren

[Oaktech]

Their status page says incident acknowledged and that it is responding slowly, which is what i'm seeing - every action I've tried to perform this morning since 8:05 has succeeded, but it's taking between 30 seconds and a minute to do everything.

[Olliedawg]

I'm able to login, however site is basically un-useable at the mo

[supportman]

Seems slow and overloaded, strange as its been really fast and responsive recently.

[michael2k6]

Our experience is that it's so slow to load it's unusable. We've also had some 504 errors and timeouts. Latest update is they're making infra changes and so it'll be unavailable for the next 15 minutes (posted 08:57 so until 09:15 ish).

[Olliedawg]

Just another manic Monday

[CHiLL]

Is this a nationwide outage or just affecting server clusters? Just wondering since they seem to describe it as a 'minor' incident.

[TheCookieMonster]

Hopefully won't last. First time it's been bad in ages though and in general system performance has been quite good for us lately.

 

Stuff like this is a reminder that we should all have backup processes in place - paper registers etc. to meet statutory obligations. Bromcom aside, never know when the internet might just pack up. Should always have a plan.

 

Teacher app also appears to still work btw, so must use a different service.

[Olliedawg]

Status has been upgraded to "Major Incident"

 

We regret to inform you that the changes made to the infrastructure have not improved the performance.

The Could team is in contact with our hosting provider for further assistance.

 

I can login, however still very very slow

[michael2k6]

It appears to be the webapp sat on the front end which is either performing not as expected or is overloaded in some way. That's hosted in Azure. The symptoms are similar to a DDOS but i'm not suggesting for a second it's malicious, just that it's a similar symptom. Not sure if they've got cloudflare infront of it or anything.

Hadn't thought to use the teacher app, but you're right in that it's a different front end service so that's probably unaffected - it'd have been nice for them to suggest that as a workaround though wouldn't it.

 

The reason it's classified as a minor incident is because the status is public and the reality of the situation is likely to be devastating to their reputation in the market. Unfortunate reality is that with so few players and such rapid growth as people move away from SIMs there's little to no motivation to do a good job of it. I wouldn't normally be this blunt or vocal but the amount of issues we've had since i've been in post only 18 months ago (not just full outages but things like reports timing out or taking forever and partially completing), I've run out of care for it now. I've staved off the SLT enquiring about a move to Arbor for quite a while but this will likely take the decision out of my hands.

[Linfit]

Just had a further update:

 

Known Issue - Bromcom MIS Performance - 20/05/2024 - Update

Good morning,

Apologies for the ongoing impact that you are experiencing with Bromcom MIS.

Our engineers have opened a critical incident case with our cloud services hosting provider and are currently investigating why we have experienced a failure in our systems this morning.

Together with the hosting provider, we are working as quickly as possible to restore the service.

Once resolved and we understand why this happened, a root cause analysis (RCA) will be provided.

 

Thanks

 

Ours is working now but response is slow = but we have been able to to get cover sorted and start doing registers.

[TheCookieMonster]

Ahh so sounds like it's infra related rather than software. The status page has also been pushed up to 'Major'.

[mthomas08]

Latest email we got is it's tied to their performance upgrade improvement but made it worse.

 

Normally I would be critical of Bromcom but the last 6 months have seen a huge improvement to reliability that rivaled SIMs. Obviously, Sims was reliable because I went out of my way to make it reliable and avoid doing any work within working hours and their updates installed during holidays only...

[supportman]

Appears to be all fixed and super fast again

 

Like others have said, its been great the last 4 months.

 

Seems like it was a scaling issue at Azure this time.

[garbage46]

I am reluctant to say it out loud, but Bromcom now appears to be more responsive and faster than we've ever experienced

[Dr_Zeux]

I am reluctant to say it out loud, but Bromcom now appears to be more responsive and faster than we've ever experienced

 

What about running reports? Ours are taking forever to run!

 

 

Update at 1pm - they seem much faster and more responsive now. I think they've resolved the issues from this morning...

Edited May 20, 2024 by dbluston

[Steve21]

Appears to be all fixed and super fast again

 

Like others have said, its been great the last 4 months.

 

Not sure who's been saying that! It's been terrible since we've signed up for it!

 

Steve

[supportman]

Not sure who's been saying that! It's been terrible since we've signed up for it!

 

Steve

@mthomas08 said it above.

 

We've also had no speed issues in recent months. Its been very responsive.

 

I wonder if you are still on the old slow hosted servers and perhaps some users have been migrated to the saleable hosts perhaps? Just guessing...

 

Reports are not the fastest but that's the same for all MIS's generally. More to do with amount of data and report design rather than server power I would think.

[Dr_Zeux]

Not sure who's been saying that! It's been terrible since we've signed up for it!

 

Steve

 

I feel your pain Steve, we've had so many issues with Bromcom since signing up, and no one seems to care - especially the so called 'Customercare@....' I could give you a list a mile long of things that have been so poorly dealt with, a number of complaints i've had to raise, bugs in their software, development ideas, assessment system nightmares, their limited resources, etc etc, but my time is now worth more and I can no longer waste my energy on Bromcom...