[sims] SIMS .net Autumn Release pulled

[vikpaw]

If only 3/1000 schools are affected, perhaps pulling the software was an over-reaction. Perhaps it could be left up but with a warning that xyz function doesn't work, and schools can then make a decision themselves on whether or not to proceed.

 

I think in this instance, it's a good job it's been taken out of action. If they told us what didn't work, which they have now, you wouldn't take it anyway. Registration groups / pastoral structure is affected, and you can't know exactly what's been happening in the history of your SIMS, so you might be affected and not even know.

 

They cant win either way really can they. some will want it now now now and others will be of the wait and see type.

I would say that it does seem a very small minority and perhaps on this occasion they might have left it up.

 

Does anyone test a version of the updates on a mirror/non live environment but with their data? i've used a VM a few times to test various things and wondered if anyone has the time to do this a matter of course?

 

I do this every time, without fail. It's almost a no brainer now. I can't test everything, but i always download the ISO and put it on a virtual test server. I haven't done it yet, because i also wait for the issues to crop up, as downloading another copy will take us a while, so would rather it was the patched one we install.

 

You can't test everything, but i just do it to see if it actually runs the upgrade files without conflicting with anything else. The basics work, i.e. assessment / reports, registers and data reports. Day to day stuff. I don't test in detail, but if it seems to work, at least there are no show stoppers. Last time the homepages went funny, but it didn't affect my test user so its still difficult to be sure. At least i know that if i work late or at a weekend and have told all users to be out of the system, it will at least update the db and not be a total waste of time even if some little bugs do creep in.

 

Now that I seem to look after 2 schools, 1 school will be sighing in relief, the other is happily managed by someone else when it comes to SIMS. I'm glad that Capita have the ability and desire to pull something like this. The knock on effect is disastrous even at a rate of 3/1000. Kudos to Capita for acting quickly and pulling it I say.

 

I'm with you there, we're looking at possibly 66 schools in total out of their 22,000. Not bad really, but you can't be sure. I think it could affect quite a lot of schools.

 

I wouldn't have known had i not seen this thread. Yes i'd have looked at SupportNet eventually, as i have a few forums to post replies in, but it's not obvious. Every single item in the news is highlighted. All 12 have something that needs attention. 6 New, 6 updated. This really should be bigger, flashing, or something.

 

The last email i got, was on wednesday just telling me that SOLUS 2 was updated with the CENSUS patched release.

 

I still think they need to work on the communications. Not a thing on twitter, nor an updated email. Nothing on the other new websites. It might seem like bad marketing to publicise a fault, but seriously, i think we'd all agree, it's damn good marketing for us, the current users.

 

I'm updating the main Sticky Autumn release thread with more details of the actual fault, as posted on SupportNet. http://www.edugeek.net/forums/mis-systems/85606-autumn-release-2011-a.html

[Sivadam]

...Does anyone test a version of the updates on a mirror/non live environment but with their data? i've used a VM a few times to test various things and wondered if anyone has the time to do this a matter of course?

 

When I attend UAT then I test the forthcoming upgrade against my school's own dataset. That is what UAT is aimed at. Capita want to test against as many different datasets as they can. That is why they also release beta versions to LAs and approved test schools! UAT also gives you an opportunity to feed your own ideas into the system so you can, even if only in a small way, determine how the software moves forward.

 

Ironically, I have applied for January's UAT to test the Spring 2011 Release and the session is full! It seems that more and more users are seeing the benefits of UAT. I would like to attend but maybe this is the time to get the Beta for that release to test on my new SIMS Test laptop. It won't be the same as attending UAT though!

 

As for not having time for testing, I also use it for my own training needs! If new functionality is being released with a future release then you can trial that part of the software at UAT! This saves time (and possibly money) later, as you would most likely not need further training in that new functionality!

[PhilNeal]

A quick update. I've been into see our service desk and can confirm that we've had 5 cases where the curriculum structure has been changed incorrectly. We suspect however that although 1000 downloads have taken place not that many schools have actually applied it as we'd have expected more. We're investigating what happened as this is clearly unacceptable but the top priority is providing a fix for schools impacted and getting the release out again.

[pete]

Does anyone test a version of the updates on a mirror/non live environment but with their data? i've used a VM a few times to test various things and wondered if anyone has the time to do this a matter of course?

 

Yes. We have a test SIMS environment because we remember the Bad Old Days. Now it's externally accessible SIMS downtime affects more than the people within school at $time_it_breaks.

 

Besides, having to answer "why is SIMS broken?" with "because I didn't test the latest version" is embarrassing.

[enjay]

You can't test everything, but i just do it to see if it actually runs the upgrade files without conflicting with anything else. The basics work, i.e. assessment / reports, registers and data reports. Day to day stuff. I don't test in detail, but if it seems to work, at least there are no show stoppers.

I disagree with that so much! Accepting that complete testing is not possible, either by us or by Capita, the basics you list are something which capita can test. Something deep in the bowels of SIMS not working is understandable, possibly even excusable, however problems with fundamentals like class lists and pastoral structure are simply unacceptable.

[matt40k]

I disagree with that so much! Accepting that complete testing is not possible, either by us or by Capita, the basics you list are something which capita can test. Something deep in the bowels of SIMS not working is understandable, possibly even excusable, however problems with fundamentals like class lists and pastoral structure are simply unacceptable.

 

re-read the post.

Who is affected:

 

Customers who have Registration groups which

have been in different year groups in the past. This could be caused by a number

of reasons including:

 

1) At some point in the past the school have

amended their pastoral structure to include a mix of Horizontal and Vertical

Year groups. The registration groups would therefore have moved years from Year

1/Year 2 to Year 2 for example.

 

2) The school have decided to rename a

registration group and assign it to a different year. In other words, instead of

creating a new registration group for someone they have renamed an existing one

and moved its year.

 

 

 

What effect does it have?:

In SIMS it will appear as though a new Year has been created (for example Year

1/Year 2) and it will appear to take some of the Registration groups with it.

 

 

To be fair to Capita, they're now going to test this scenerio in every future build

[synaesthesia]

Obviously it's no surprise. When was the last time a perfect piece of software was released to the public? Only when entire games/programs occupied less pages of code than you can count on both hands and were on certain rubber-keyed computers.

Capita have to be particularly careful though; unlike a game or version of Office, a tiny flaw in their MIS system could cause major problems that have an impact on school's finances, resouces, staffing - just about anything and everything. I think we'd all agree they are better to pull a release until such problems are solved rather than waiting to patch it, if that's even possible. There will be very few people in the country that have as close a grasp on this as Mr Neal here, so despite my dislike of Capita as a company will happily believe what he says is for the best.

[vikpaw]

I still stand by my statement, you can't test everything. Mostly issues are related to nuances at particular schools. Everybody does their timetable differently, has different reports / subreports, changed their pastoral structure / didn't change their structure, you can't test all of that. Things a school has done wrong in the past can have an effect on future usage. That's why there are so many site specific patches. A school shouldn't have to, but is the only person who really can do a thorough test.

 

@Sivadam can attest to the fact that his school finds more faults than any other, because of the level of usage they have with SIMS and fine tuning that's been done.

 

To be honest, i think Capita should hold more school data and be allowed to test on it. I've offered mine but it's not allowed. I used to keep a copy of my data there regularly so it's ready for testing if i do get an issue. When there is an issue the last resort is for them to look at your data, i don't see why they can't keep it from schools who give permission, run an anonymiser on names / birthdates and phone numbers if they have to. so they have more real test data. Not all schools can take part in UAT, but if capita did it for you, then you'd have more confidence in it.

 

For our part, we really have to do at least basic testing. We do so much in SIMS now, we couldn't live without it for even a few hours. That's even before we make it available externally!

 

In an ideal world the updates would be backwards compatible to the point where you could rollback an upgrade if it went funny, but keep any new data that may have been recorded.

[apeo]

I disagree with that so much! Accepting that complete testing is not possible, either by us or by Capita, the basics you list are something which capita can test. Something deep in the bowels of SIMS not working is understandable, possibly even excusable, however problems with fundamentals like class lists and pastoral structure are simply unacceptable.

 

Isn't this abit of a cheeky comment when a couple of posts back is:

 

No, for three reasons:

1) time

2) can't test everything

3) shouldn't have to. I don't do that with any other company's software.

[enjay]

I don't see why you think it is cheeky - in both posts I state that I believe that Capita should test the software before release.

[PhilNeal]

We do test the software endlessly but sometimes errors are invisible until they get out into the field. Every release goes through unit, system, UAT and advanced release to try to get it right. I'm afraid that humans will always make errors but we do everything we can to double and tripple check. I'm very concerned that these errors occurred because there was clearly a process failure - in this case the error is obvious when it occurs but it doesn't go wrong that often so it's difficult to spot unless you have that condition in the data.

[apeo]

I just find it funny that it was in reponse to someone who does test it despite the fact that you dont test it and here is what you state:

 

"I disagree with that so much! Accepting that complete testing is not possible, either by us"

 

It might just be me but i find it cheeky to state how much or how little someone should do testing prior to deployment especially after stating '2) can't test everything' in ref to what you test.

[vikpaw]

No matter. We all probably read it differently.

The point is the more of us that test the better it is.

Is it possible @PhilNeal to release an ISO of the beta so you those schools with test setups can volunteer to do some optional testing, outside of any formalised process. Any heads up on a problem is better than nothing.

[Rawns]

At our LA, we always build a couple of new VM images and upgrade SIMS/FMS to the Beta when it's released, and then then we also build more VM's and upgrade them to the main release before we roll it out to Schools. The VM's are always available to the team so that each officer can test their own area supported areas for any issues. As well as the VM's, We also have a number of different SIMS test setups where different Schools types have allowed us to collect a copy of their data. We use this for testing new releases as well as testing instructions that we are producing to ensure that they are accurate.

[enjay]

I just find it funny that it was in reponse to someone who does test it despite the fact that you dont test it and here is what you state:

 

"I disagree with that so much! Accepting that complete testing is not possible, either by us"

 

What I meant by "us" was schools in general, not specifically the people at my school.

[matt40k]

If you had a central hosted environment you could have two copies of all the schools data, one live and one beta\training\next release. Ideally you would only want to give this access to certain people however otherwise they could end up entering data onto the wrong system!

[fiza]

Just posted on Supportnet

 

*** SIMS 2011 Autumn Release - Please Read 28/11/2011

 

28/11/2011

 

We are continuing to investigate possible resolutions for the data issues described below. Work on producing the new SIMS 2011 Autumn release is also continuing. Unfortunately I cannot as yet announce a confirmed date for them. I am hoping to be able to give you more information tomorrow.

 

We apologise for any inconvenience that this delay might cause you but please be assured that we are doing all we can to resolve this.

[Sivadam]

I would like to refer SIMS Users to the following link from the Sticky Items on the SupportNet website.

Apologies to those that do understand how much testing of SIMS Releases is done.

Those that don't, may I humbly suggest you read the document.

 

The link is: User Acceptance Test

Edited November 28, 2011 by Sivadam

[vikpaw]

Out of interest, do all LAs / Support companies take part in Beta testing? You'd think it was a must if they are supporting more than even one school.

[Rawns]

Out of interest, do all LAs / Support companies take part in Beta testing? You'd think it was a must if they are supporting more than even one school.

 

I thought the Beta is made available to every LA in Schools Manager under the Tst1 and Tst2 sites? Seems like a no brainer to install it on to a virtual test system. Capita provide test data if required, but I'm sure if a school is happy for you to collect a backup, there is no reason why you can't test it against a "live" schools data.

[matt40k]

Out of interest, do all LAs / Support companies take part in Beta testing? You'd think it was a must if they are supporting more than even one school.

 

We tend to only do it if they're is something in the pre-release notes that interest us, such as FMS or those cool new features. The Beta doesn't always include everything from the final release either and we tend to only have our's\Capita training databases which we can test on. So it's not perfect.

[vikpaw]

If i was with an LA i would probably insist they tested it on my data. In the end it's going to save them hassle supporting me later

less calls from me = happier supportline monkey

[matt40k]

If i was with an LA i would probably insist they tested it on my data. In the end it's going to save them hassle supporting me later

less calls from me = happier supportline monkey

 

I ain't got the time to backup 350+ schools SIMS and FMS databases, upload to our network, then restore, apply the update, then check them. Maybe if everyone was centrally hosted. If school wants to setup a test environment, I'm fine with that, but doing it all for them, that would be mega £££.

[creese]

Out of interest, do all LAs / Support companies take part in Beta testing? You'd think it was a must if they are supporting more than even one school.

 

I wish we had the time and resources.

[Sivadam]

I ain't got the time to backup 350+ schools SIMS and FMS databases, upload to our network, then restore, apply the update, then check them. Maybe if everyone was centrally hosted. If school wants to setup a test environment, I'm fine with that, but doing it all for them, that would be mega £££.

I wish we had the time and resources.

... and that is where little old me comes in (and a few others! ) with UAT!

 

UAT is well worth the time and effort - especially when new functionality is being delivered by a future release that you are testing!

Edited November 29, 2011 by Sivadam