Schools Week - Arbor’s Kickbacks

[supportman]

And this, ladies and gentlemen, is why lift and shift is a bad approach for could services.

 

Don't agree with that at all. We had far more downtime from Capita Sims with "update patches" than any cloud SASS service downtime.

 

I'm so much happier knowing someone else is panicking when a service doesn't run

 

The most important thing in any downtime incident is the company learns the lessons and makes sure it doesn't happen again.

[Primus]

For a start - a change was requested to uplift resource allocation on August 26th, it wasn't approved until the early hours of Sunday September 1st - this is bizarre. I'm going to assume that Bromcom must be working across multiple timezones but first of all why the delay and second why would you make changes so close to the start of an academic year. The increased demand of the start of an academic year is well known and established, surely making changes in the early hours of a Sunday is bad time to be doing this.

[dmj]

The most important thing in any downtime incident is the company learns the lessons and makes sure it doesn't happen again.

 

I'll try and explain:

Holding CAB meetings for '100% increase' on resources by way of a manual change is how we used to do it in the olden days: It's a lift and shift approach.

In modern web based companies we dynamically scale the application and it's databases based on metrics (load, CPU/RAM, available workers, traffic).

This way we constantly test our ability to scale and can mitigate when things go wrong eg scaling to other regions etc. Scaling should be a normal thing that happens automatically without manual intervention: My servers scale around 300% DAILY without downtime* It saves a huge amount of money not running all the machines when they are not at capacity. Everyone is happier: less engineering time for adhoc scaling, cheaper product for customers, more responsive when it's at peak usage.

 

bromcoms takeaway is that they will carry on as planned, but make sure the resources are available with prior agreement.

My takeaway is that they need to move on from the old style and that a re-architecture will save them money, improve their reliability and allow the company to scale.

 

* Disclaimer: I work for a slightly (but not hugely) larger company that deals with web based apps, and part of my role is to scale and monitor the applciation load.

[FN-GM]

[emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]][emoji6[emoji640][emoji637]][emoji637][emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji637]]][emoji[emoji6[emoji640][emoji638]][emoji640][emoji640]][emoji639]]I'll try and explain:

Holding CAB meetings for '[emoji637][emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]][emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]]% increase' on resources

 

They described it as a standard change so shouldn’t have gone to CAB in the first place.

[FN-GM]

For a start - a change was requested to uplift resource allocation on August 26th, it wasn't approved until the early hours of Sunday September 1st - this is bizarre. I'm going to assume that Bromcom must be working across multiple timezones but first of all why the delay and second why would you make changes so close to the start of an academic year. The increased demand of the start of an academic year is well known and established, surely making changes in the early hours of a Sunday is bad time to be doing this.

 

I read this as it was approved the be executed on 1st September and actually approved on the 26th.

[dmj]

They described it as a standard change so shouldn’t have gone to CAB in the first place.

 

Yeah, did spot that. I've seen some ITIL processes direct to CAB for standard changes if they have budgetary impact- so figured it was one of those.

[mavhc]

Isn't the point of being a cloud native company that your resources are dynamically assigned?

 

Of course tying yourself to one cloud provider in one region is the first mistake

[FN-GM]

[emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]][emoji6[emoji640][emoji637]][emoji638][emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]][emoji[emoji6[emoji640][emoji638]][emoji640][emoji6[emoji640][emoji638]]][emoji637]]Yeah' date=' did spot that. I've seen some ITIL processes direct to CAB for standard changes if they have budgetary impact- so figured it was one of those.[/quote']

 

The way I look at it, a standard change is routine and is the norm. So if it did have a budget impact it wouldn’t be a standard change anymore as it’s outside of the norm.

 

From the outside looking in, doubling capacity is quite a big jump. I wouldn’t class that as a standard change but a normal change.

Edited October 8, 2024 by FN-GM

[psydii]

They requested more capacity from the cloud infrastructure provider and did not get it, not only did they not get it, but a process failure allowed (some) of the previously provisioned lower tier capacity to be recycled back to Azure before the new capacity had been confirmed.

 

Capacity constraints in Azure are not unheard of.

[mthomas08]

Given the reliability and capacity issues we keep seeing reported on here (which will only be a small sample), I think that is very arguable.

 

I don't know.. every time we had a SIMs issue I never once came on here to see if other schools went down..

Every single time 'reliability' pops up here.. I remind staff about how unreliable SIMs was.. AND the fact I would often do work outside of normal working hours to KEEP sims up and running.... perhaps I should have installed those patches during working hours.. and it would be a long list of patches...

 

Like staff went very quiet when I said we are looking else where - their complaints were daily... DAILY about SIMs..... constant crashes.. freezes.. crashes.. more crashes.. random MEMORY errors..

 

I will add if SIMs was built on a portal 10 years ago instead of internally hosted.... I don't have faith that it would be this magical reliable system..

 

Can't speak for Arbor.. but SIMs was awful... AWFUL.

Edited October 8, 2024 by mthomas08

[psydii]

How do Bromcom and Arbor handle user actions that touch a lot of records? For example applying a new timetable mid-year, or compiling the census report? These are the two actions for us that would regularly cause SIMS to momentarily come to its knees. Mostly solved here by people learning to perform certain tasks off-peak (which is a pretty standard technique when doing intensive tasks on a database that doesn't have a 24/7 global foot print). Are these actions ok to perform in hours?

 

The other thing that used to bring SIMS down were some 3rd party data sync tools being too aggressive. By the sounds of it this term ESS have done exactly this to themselves for some of their users. For us a big one was the Document Scanning system - it was way to keen to check sims for changes so its UI was always up to date. I imagine though neither Arbor or Bromcom have scan-to-cloud MIS integrations at all, and carefully rate-limit 3rd parties calling their APIs?

[AlistairB1983]

How do Bromcom and Arbor handle user actions that touch a lot of records? For example applying a new timetable mid-year, or compiling the census report? These are the two actions for us that would regularly cause SIMS to momentarily come to its knees. Mostly solved here by people learning to perform certain tasks off-peak (which is a pretty standard technique when doing intensive tasks on a database that doesn't have a 24/7 global foot print). Are these actions ok to perform in hours?

The other thing that used to bring SIMS down were some 3rd party data sync tools being too aggressive. By the sounds of it this term ESS have done exactly this to themselves for some of their users. For us a big one was the Document Scanning system - it was way to keen to check sims for changes so its UI was always up to date. I imagine though neither Arbor or Bromcom have scan-to-cloud MIS integrations at all, and carefully rate-limit 3rd parties calling their APIs?

Can't speak to Bromcom, but as far as Arbor is concerned it has no knock-on effect to other users. Occasionally, during particularly busy periods or if you have made a large number of requests, it may process them in the background - meaning you can't instantly see the changes you've made. But they do always appear, and it has no impact on other users whilst it's running in the background.

 

We've been running the census the past week or so, and no one has complained once that the system had slowed down for them.

[supportman]

Can't speak to Bromcom, but as far as Arbor is concerned it has no knock-on effect to other users. Occasionally, during particularly busy periods or if you have made a large number of requests, it may process them in the background - meaning you can't instantly see the changes you've made. But they do always appear, and it has no impact on other users whilst it's running in the background.

 

We've been running the census the past week or so, and no one has complained once that the system had slowed down for them.

 

Pretty much the same with Bromcom, never noticed any "system overload" type behaviors like we did with Sims. Our census is almost transparent now and very easy to do

 

Some parts of the system are slower, such as complicated reports and the attendance pages but I think that's more to do with the view layout than anything else.

[mavhc]

You put it in the queue. The best solution involves: 1. being able to see the queue, you know that it's still processing and will be done in maybe 1 min/hour/day. 2. being able to adjust priority of the queue. 3. some way to show that data you're viewing now is in the process of being updated.

 

Related: I hate any program that shows you default data instead of "waiting", "there are no results" -- arrgh my entire database has gone. 2 seconds later the results load.

[psydii]

Related: I hate any program that shows you default data instead of "waiting", "there are no results" -- arrgh my entire database has gone. 2 seconds later the results load.

 

At the risk of irritating DJM, this is mostly a cloud thing. It seems to stem from the fact the hyper-scale databases are only best-effort consistent and this mentality (somewhat necessary at global scales) has permeated into developers and product managers operating at the front-end customer level. As a small/medium/single geography entity we expect/need the UI (and reports) to show consistent, accurate, current data... but cloud principles mean that for the back end this is expensive/impossible.

 

You see this a lot in mobile apps where the additional pressures of battery-life management and unreliable connectivity reinforce the approach of show stale data and update the view as-and-when the data becomes available.

[mavhc]

It's not even slow data though, it's no data, you're showing the wrong data instead of saying you haven't got the data yet, that's infinitely worse

[dmj]

Sounds like a caching issue than DB to be honest. The amount of places things are cached these days is crazy.

[mavhc]

I'm talking about a specific design, where you show the wrong data on purpose until the correct data loads "you have no emails" "you have 0 files" etc, instead of showing nothing, or a "loading data" symbol

[psydii]

"loads fast but might be wrong until such time as we can get the consolidated view of the data ready for you at which point we update and what the user sees completely changes" whatever the underlying technological reasons, is pretty poor ux and only in the cloud era did it start to effect everything.

 

That said, AD has the same underlying problem for similar reasons, it can present you stale data until both the server you are connected has the latest update *and* you remember to manually refresh the view. Usually people aren't making massive changes in AD though, so its less likely to induce the "oh my god where has all the data gone" moments.

[Damon]

Ooof! Well this thread has taken off a bit. 5k+ views and now a sticky. Guess it's touched a nerve with folks and I can understand why. Could be why someone posted up a NotebookLM podcast of it:

 

 

How is NotebookLM so damn good at creating these things! I find podcasts a good way to digest a lot of info. I wonder if Edugeek could use it to create weekly updates!

[Steve21]

Ooof! Well this thread has taken off a bit. 5k+ views and now a sticky. Guess it's touched a nerve with folks and I can understand why. Could be why someone posted up a NotebookLM podcast of it:

 

 

How is NotebookLM so damn good at creating these things! I find podcasts a good way to digest a lot of info. I wonder if Edugeek could use it to create weekly updates!

 

Unlisted Video

Channel Made 3rd October

 

Right... Come on lol

 

Think you're misunderstanding where the "touched a nerve" actually lays on and it isn't Arbor

 

Steve

[Damon]

Haha well it's a tough crowd eh! The pod link was from linkedin so maybe just a throwaway account to upload it on the platform without issues?

[Steve21]

Haha well it's a tough crowd eh! The pod link was from linkedin so maybe just a throwaway account to upload it on the platform without issues?

 

Well you are a representative of Bromcom who's PR drone made this stupid thread in the first place... Maybe if Bromcom staff spent as much time fixing the issues that people are raising rather than doing these kind of posts there wouldn't be so much grief when they do post something that's helpful towards the community?

 

It really does say a lot when more time is being spent trying to tell people why not to pick competitors than promoting why Bromcom is the company you should be selecting

 

Steve

[Damon]

Yikes ok well I'm not sure what representative means to you but I'm a consultant so work with multiple companies including Bromcom as per my signature. I'm not on the payroll but work with them on various projects as and when the needs arise. Just like the migration challenge as I can be the independent judge along with a well known industry bod who used to work at ESS. I also work for a big support team that supports Arbor and SIMS too. I spent years at an LA doing the same thing and whilst I can see where you're coming from to an extent, regardless of my links to Bromcom I'm against the concept of paying cash to a public body based on an incentive scheme. Even ESS don't do that!

[Steve21]

I'm not sure what representative means to you but I'm a consultant so work with multiple companies including Bromcom as per my signature. I'm not on the payroll but work with them on various projects as and when the needs arise.

 

So maybe I'm misunderstanding then, you're not the Bromcom Damon that:

* Has a Bromcom email address

* Has dozens of pages/blogs on Bromcom websites

* Has been their "Marketing Manager"

* Has been their "Product Owner for UI/UX"

* Has worked on Bromcom stalls at Bett/London Acadmies etc

* "Insert More but I hope you get the point"

?

 

Guess you have a very similar imposter around then! sigh

 

Steve