[bromcom] Outages

[Strawdog]

Anyone else considering moving MIS due to the large number of outages since the summer?

 

We had a meeting with a couple of their senior staff due to the massive disruption throughout September - they were candid, assured us they had learnt a number of hard lessons and gave credible explanations as to how they were going to improve things in the future (better communication, increased resiliency, more receptive to change requests, etc.). Fast forward to this week and we had an extended outage on Monday afternoon, which I've still not received an explanation for, and so far it's been down 3 times today...

 

We'd be royally screwed in the event of a fire or an Ofsted visit :\

[Strawdog]

4 times

[6Foot2]

It was only yesterday that our Leadership Team confirmed our move from Serco Facility to Bromcom to go live in February next year. This will be externally hosted initially, to be moved in-house if it proves to be too slow.

 

[Strawdog]

The speed is usually fine (we originally had an onsite server running it but moved to the cloud platform a couple of years back) and generally, apart from them being a bit tardy with implementing valid customer improvement suggestions and the usual quirks any platform has (along with those staff who hate any and all change), it's been very good for us. Since September though it's been flakey with large numbers of outages.

 

It appears they don't/didn't have adequate safeguards to stop some users eating up all the cloud server resources (they had some customers hammering their API with constant requests) back in September along with a hardware failure at their host (SAN controller died taking down the entire SAN even though it was supposed to have failover) which killed all their VMs. Not sure what the current issues are as we haven't been told anything beyond 'they're investigating'.

 

Bottom line is SLT don't care about why it's down, or whose fault it is, they just want a stable platform and for the past couple of months we haven't had that - I have confidence Bromcom will sort it out as they have some very good techs, and I'm loathe to give money to Serco or Capita if we felt we had no choice but to move, but they need to do it quickly otherwise their customers are going to lose faith and I'm going to have a new 'fun' project on my desk to find a new provider

[Bromcom_Jay]

From Bromcom's support front page:

 

"Bromcom has been vigorously working on reviewing and improving our SQL database solution to eradicate the risks we experienced with the October outage. These have been progressing successfully.

 

The slow performance experienced yesterday was due to an SQL indexing problem that our tech department and the team at Microsoft are working to resolve. Whilst it did not affect all our cloud customers, we are now aware that many were affected. Overnight we have returned SQL indices to their state before we had performance problems and have re-indexed. So far system performance is performing without any issue and we have no reason to believe this will change.

 

Rest assured that our performance monitoring is on-going and we are working to enhance the monitoring and to publicise the cloud status on our support site.

 

On behalf of all the team at Bromcom we sincerely apologise for any interruption and inconvenience this may have caused our users, but our aim is to strengthen the cloud and its performance to ensure this doesn’t happen again.

 

Simon Lewin

Head of Customer Relations"

[pcstru]

We self host our Bromcom MIS, a choice that Bromcom isn't very keen on but which seems to pay for itself in a number of ways. In general our users really like the system so it is a bit of a worry if they have hosting problem and are not addressing them competently as in the end that risks the business and consequently a risk to us as a customer (even if we aren't experiencing the problems!).

[Ephelyon]

We're looking into moving to Bromcom and self-hosting it at the moment. Do you know if it's significantly more expensive to do so?

[pcstru]

We're looking into moving to Bromcom and self-hosting it at the moment. Do you know if it's significantly more expensive to do so?

It was significant enough for us! More in up-front cost (~£10K on purchase) than year on year support & updates (~£2K) but it affects both. It is still competitive with the competition in terms of self hosting. Your costs will be different; only way to know is to get a cost from them (perhaps you can negotiate them down a bit given the reliability issues with the cloud product!).

[Ephelyon]

Very true, and many thanks for that. That is pretty significant... is that with them providing hardware as well or something?

 

I was just going to run it on our own infrastructure. I currently have an IIS VM and a SQL VM, both with 4 cores and 16GB RAM, to run PARS Connect with adequate performance. Is that similar to what you've got, assuming it's on your own infrastructure?

[pcstru]

Very true, and many thanks for that. That is pretty significant... is that with them providing hardware as well or something?

No. Nor did it include DB licensing etc.

I was just going to run it on our own infrastructure. I currently have an IIS VM and a SQL VM, both with 4 cores and 16GB RAM, to run PARS Connect with adequate performance. Is that similar to what you've got, assuming it's on your own infrastructure?

We allocate 16 cores on a 32GB server which is running both the DB and the web services. Performance is adequate for our demand of ~200 users (max concurrent). I run some SQL directly against the DB and it seems to hold up well even at busy times.

[Ephelyon]

Okay, many thanks for this info

[Michael]

No. Nor did it include DB licensing etc.

 

We allocate 16 cores on a 32GB server which is running both the DB and the web services. Performance is adequate for our demand of ~200 users (max concurrent). I run some SQL directly against the DB and it seems to hold up well even at busy times.

 

How many actual sites is that serving? A server of that specification must have cost a small fortune!

[pcstru]

How many actual sites is that serving? A server of that specification must have cost a small fortune!

It serves one site - although we are quite large. It's not that unusual as a server spec (although IMO companies often drastically over spec to try and avoid issues). The underlying host hardware is a Dell poweredge R730xd with ~160GB RAM, two Xeon 8 core processors and 8Gb of network bandwidth split 4/4 between our the main and storage networks. It's also running our production Sharepoint server and the head of our main file server. If there are bottlenecks they relate to the number of spindles at the back end, not RAM or CPU. Server hardware was, ISTR around ~£4K maybe a couple of years ago.

[Michael]

It serves one site - although we are quite large. It's not that unusual as a server spec (although IMO companies often drastically over spec to try and avoid issues). The underlying host hardware is a Dell poweredge R730xd with ~160GB RAM, two Xeon 8 core processors and 8Gb of network bandwidth split 4/4 between our the main and storage networks. It's also running our production Sharepoint server and the head of our main file server. If there are bottlenecks they relate to the number of spindles at the back end, not RAM or CPU. Server hardware was, ISTR around ~£4K maybe a couple of years ago.

 

Very interesting - but presumably if you ran it at multiple sites, you'd opt for the hosted version?

[pcstru]

Very interesting - but presumably if you ran it at multiple sites, you'd opt for the hosted version?

No. The MIS is too critical day to day and cloud hosting presents a number of issues that I'm still not comfortable with for critical systems (and we seem to do better anyway on the basics - day to day availability!). An important one of those is being in control of change and I don't believe suppliers have recognised or addressed that problem (so for instance, the last major overhaul of the Bromcom UI was released to the cloud without enough notice of the change for us to be able to fit training and awareness into scheduled training days. I just don't want to be in a position where 200 users log on and wonder where everything is, so I like to be able to anticipate change and schedule some activities to raise awareness and if necessary train people). Not being in control of updates and upgrades with cloud systems can cripple an organisations ability to plan and manage delivery of change. They often end up just throwing the change at people and letting them sink or swim (and the business along with it!).

[Bromcom_Jay]

Hey pcstru,

 

Important feedback that I haven't heard from our users (although I have only been at Bromcom 1 month). I will make sure the powers that be understand what you are saying.

 

It is worth pointing out that Bromcom has had 1 day of slow speed (16/11/16), 20mins of outage on Monday 14th (at around 3pm) and one half day of outage last month. That is everything ever - with better planning of technical upgrades and by tackling the issues we have encountered, we can't see any reason why it shouldn't continue to perform as normal in the forseeable future. I think the advantages of going cloud outweigh a couple of network issues, if you are hosted locally you could have had a half day out and some slow speeds yourself at some point in the last 5 years?

 

Keep the feedback coming though!

 

Jay

[Michael]

I think the advantages of going cloud outweigh a couple of network issues, if you are hosted locally you could have had a half day out and some slow speeds yourself at some point in the last 5 years?

 

I agree, it doesn't make financial sense and there are more pros to hosted rather than hosting yourself in my professional opinion. But that's just it, it's my opinion.

[Strawdog]

It is worth pointing out that Bromcom has had 1 day of slow speed (16/11/16), 20mins of outage on Monday 14th (at around 3pm) and one half day of outage last month. That is everything ever

 

I understand you need to paint a rosy picture Jay but it was completely unusable for large amounts of the day on the 16th, the outage on the 14th was longer than 20mins and these were both on top of the outages we had on multiple days during October. We have also had outages in previous years although nothing like the recent spate.

 

What you have written is simply not true.

[Bromcom_Jay]

it was completely unusable for large amounts of the day on the 16th, the outage on the 14th was longer than 20mins and these were both on top of the outages we had on multiple days during October. We have also had outages in previous years although nothing like the recent spate.

 

Strawdog - These are the outages I have recorded from our side, as mentioned previously, having only been at Bromcom since 14th October I don't know of all instances. Every individual school will undoubtedly have their own records and I'd encourage any school to make sure as soon as the system isn't performing it's best to flag up a support ticket and ask the team to look into it.

 

What you have written is simply not true.

It is true from the information we have, on the 16th the cloud system was still working, I understand it was due to an unusual amount of processing - reports from many schools was the system was slow, but still usable. On the 14th, the time recorded where nothing happened was only 20 minutes. We were also using the system for in house training and we couldn't access it for 20 mins, but then it was working.

 

I wasn't trying to mislead anyone - we have had issues with access, we are improving it and I would urge anyone to ask other Bromcom users for feedback, I was also illustrating similar issues could come up with locally hosted systems, which can cost a lot more and have other implications a cloud hosted system could solve.

[pcstru]

Hey pcstru,

 

Important feedback that I haven't heard from our users (although I have only been at Bromcom 1 month). I will make sure the powers that be understand what you are saying.

Thanks but it appears not to have been successful. We have essentially been told that we can't be appraised of future developments (a detailed roadmap) in a timely manner because often even the developers are unaware what features will not be worked on until a few weeks before release. That kind of thing sets the alarm bells ringing in my head.

[Bromcom_Jay]

Thanks but it appears not to have been successful. We have essentially been told that we can't be appraised of future developments (a detailed roadmap) in a timely manner because often even the developers are unaware what features will not be worked on until a few weeks before release. That kind of thing sets the alarm bells ringing in my head.

 

I have passed this feedback on and have gathered some more information - at all of our user group meetings, the customer support homepage and during the free monthly webinars we will be updating users on future updates and what we are working on currently. The next update - cloud or local - has an update to one module which we are asking users when they would like to update, not just letting them try to get on with it!

 

We have had a personnel change in the software development team and the road map has been neglected. I will have one in the next week and will ensure all users can find it on the support site.

 

As I am new to the company I am endeavouring to get my head around the current way we do things, but also offer insight on how to move forward - feedback from edugeek is definitely passed on and so is all feedback from users - so keep passing information on.

 

pcstru - you are welcome to PM me if you have any specific queries, as is any user on edugeek - I can't promise I will know everything straight away, but I can find the answer!

 

Jay