Problems this morning with Bromcom?

[5tu]

what a sh1tsh0w - I'm waiting for the "we should have stayed with SIMS" feedback from staff

[kevin_lane]

yea we have very limited intermitting service, currently on hold

[synaesthesia]

I've always been fairly anti-cloud, this certainly doesn't do much to put that to rest

I think for certain mission critical systems a local cache wouldn't go amiss. (I can hear the anti-server brigade screaming at that idea!)

[TheCookieMonster]

4 minutes ago, k-strider said:

They said they were restarting it at 10.30 - so probably still happening

[psynegy]

Just now, synaesthesia said:

I've always been fairly anti-cloud, this certainly doesn't do much to put that to rest

I think for certain mission critical systems a local cache wouldn't go amiss. (I can hear the anti-server brigade screaming at that idea!)

We were looking at a schedule to send the days registers to an email address each morning as a failsafe. Naturally there was an issue setting that up, but I believe they said they resolved that - so I should try again...

[pete]

If the issue is diminished capacity (which it appears to be if they've lost a server), then kicking everyone off by restarting isn't a solution.  That's just redistributing the available capacity on a first-come, first-served basis until it's gone again.

[dmj]

6 minutes ago, dbluston said:

I totally agree - this is the point I was making to my Helpdesk ticket yesterday!  How can 1 server developing a fault cause this kind of impact??  Surely, there are multiple servers that simply take over, and balance the load.  Why are such a big organisation like Bromcom, who host thousands of schools, so reliant on 1 server???

Yeah, I just work on a niche retail site and we regularly load test scaling to thousands of 'user journeys'. This is effectively national infrastructure.I think they need to get some SRE/platform engineers on this. 

[Alis_Klar]

12 minutes ago, dmj said:

Yeah, I just work on a niche retail site and we regularly load test scaling to thousands of 'user journeys'. This is effectively national infrastructure.I think they need to get some SRE/platform engineers on this. 

I don't mean to pour scorn on Bromcom at a hard time of year but their entire platform seems very badly architected where you can tell loading a simple page of names is doing thousands of database requests and is just not efficient at parsing the page.

[machy]

[synaesthesia]

And the service status page still has the audacity to read "all systems operational"

[supportman]

5 minutes ago, Alis_Klar said:

I don't mean to pour scorn on Bromcom at a hard time of year but their entire platform seems very badly architected where you can tell loading a simple page of names is doing thousands of database requests and is just not efficient at parsing the page.

 

While this downtime is more to do with broken infra (which shouldn't happen as others have point out), you hit the nail on the head about the system as a whole.

[psynegy]

Interesting to note that the partner API is still fully operational with access to school data... That would imply to me that the database is online...

[machy]

I wonder if they have been hit with the windows update issues, given they use IIS

[E_G_R2]

oh joy 502 Bad Gateway - been reported but its making Arbour look very tempting when the contract is up for renewal 

[TheCookieMonster]

25 minutes ago, synaesthesia said:

And the service status page still has the audacity to read "all systems operational"

Saw this too.. I think something automated kicked things earlier as there were two incidents reading at one point and then it deferred from Degraded to Offline and the other one went away. Seems like they're doing their best to keep people informed. We've got it hard in my schools but I dread to think of the pressure Bromcom are feeling right now. As a fellow techy I sympathise! 

[Olliedawg]

I'm fairly sure they update the service status page manually.. i can remember reading that somewhere on a previous post.

[simpsonj]

Seemingly back up here

[psynegy]

2 minutes ago, Olliedawg said:

I'm fairly sure they update the service status page manually.. i can remember reading that somewhere on a previous post.

 

Well of course! You wouldn't want to accidentally (accurately) report downtime when you can just pretend it didn't happen if you fix it fast enough...

[Olliedawg]

Back up here too, for now.

 

[synaesthesia]

I can certainly sympathise, but there are certain systems in a school which need to remain as accessible as possible. MIS and safeguarding systems being the highest priority, and even now with so many eggs in one, fragile basket we're losing access to contact information, communications to parents, attendance, medical information and more besides. If they are unable to have workable failovers, load balancing, route differencing that works then it's clear priorities need shifting. 

I seriously believe that a local cache should be implemented, it isn't that difficult of a task with minimal additional outlay or resource. 

5 minutes here and there is one thing, but to have chunks of the day offline or near-unworkable is unacceptable for such a system.

[Dr_Zeux]

9 minutes ago, synaesthesia said:

I can certainly sympathise, but there are certain systems in a school which need to remain as accessible as possible. MIS and safeguarding systems being the highest priority, and even now with so many eggs in one, fragile basket we're losing access to contact information, communications to parents, attendance, medical information and more besides. If they are unable to have workable failovers, load balancing, route differencing that works then it's clear priorities need shifting. 

I seriously believe that a local cache should be implemented, it isn't that difficult of a task with minimal additional outlay or resource. 

5 minutes here and there is one thing, but to have chunks of the day offline or near-unworkable is unacceptable for such a system.

100% agree!  Not only that, what about compensation for the substantial impact on school operations and the associated costs, given the additional resources we've had to allocate as a result of prolonged downtime and overtime that some staff have had to do as a result...  

[TheCookieMonster]

5 minutes ago, dbluston said:

what about compensation 

The  invoicing system in Bromcom Finance is pretty good.. just sayin'. 😉

[Olliedawg]

Down again???

[dmj]

3 minutes ago, TheCookieMonster said:

The  invoicing system in Bromcom Finance is pretty good.. just sayin'. 😉

They have to sell the system to bursars so it's no surprise that's where the engineering effort goes. 

 

It does worry me when the 'basics' aren't done properly, what else is getting skipped. The next JLR/M&S/CO-OP is my guess.

[Wizard101]

Migrated to Bromcom from SIMS. First 2 days school starts systems down.... what a start. Already had a phone call saying maybe SIMS was better 😅.  All we are getting are updates saying its fixed after they've only restarted the server.

From what I've been able to see and use in Bromcom which isn't much because its been down I think Bromcom has potential to be an amazing MIS, hopefully it will be better once this issue is resolved