Delays to SIMS releases

[Mcshammer_dj]

SIMS 2009 Autumn Update Release - Change of Date

 

An issue has been discovered in our final release candidate for the SIMS 2009 Autumn Update release. Due to the impact of resolving this over the Christmas period, the release date is now planned for COB on Friday the 8th of January to allow time for a complete regression test of the software.

 

We apologise for any inconvenience that this delay may cause.

Please feel free to email me if you have any concerns about this on [email protected]

 

 

Regards,

Jonathan Wood

SIMS Product Manager (Technical and Releases)

 

SO it appears that the move to seasonal release names has not really seen any improvement as the Autumn 2009 Update will now not be release until 2010!!! and COB on friday 8th in Reality means we will not be able to do anything on this until the 11th January

Edited December 11, 2009 by Mcshammer_dj

[mikeglover]

typical crapita!

[Sivadam]

Maybe some of you guys would like to ask Capita if you could take part in their UAT Testing programme, using your datasets?

 

With over 21,000 schools using SIMS it is very difficult to ensure that testing finds all of the issues in a new release, which might affect just some schools. In the past, I have found problems at UAT Testing which have not been evident in the Testing carried out prior to UAT (and there are several layers of this). Nor have they been evident to other UAT Testers. i.e. the issues discovered do not affect all establishments and, to all intents and purposes, the Update seems to be fine. It is often only when a User such as myself (and other volunteers) actually UATs the release on our own data, that these anomalies can arise.

 

On one recent occasion, our dataset would not even upgrade in readiness for UAT. This issue had not been evident on any other datasets that had been tested. As a result of finding this issue at UAT, a fix was produced when meant that the Upgrade was not delayed at all because of this and, of even more relevance, no school experienced the problem after release of the Upgrade!

 

I, for one, would prefer the delay of a Release if a bug has been found by a fellow tester that affects some, but not all, instances of SIMS. After all, we might be one of the schools affected next time!

Edited December 11, 2009 by Sivadam

[teejay]

What is involved in the UAT testing? Do you do it on your server or do you send Capita your database and they test it?

[Mcshammer_dj]

Maybe some of you guys would like to ask Capita if you could take part in their UAT Testing programme, using your datasets?

 

With over 21,000 schools using SIMS it is very difficult to ensure that testing finds all of the issues in a new release, which might affect just some schools. In the past, I have found problems at UAT Testing which have not been evident in the Testing carried out prior to UAT (and there are several layers of this). Nor have they been evident to other UAT Testers. i.e. the issues discovered do not affect all establishments and, to all intents and purposes, the Update seems to be fine. It is often only when a User such as myself (and other volunteers) actually UATs the release on our own data, that these anomalies can arise.

 

On one recent occasion, our dataset would not even upgrade in readiness for UAT. This issue had not been evident on any other datasets that had been tested. As a result of finding this issue at UAT, a fix was produced when meant that the Upgrade was not delayed at all because of this and, of even more relevance, no school experienced the problem after release of the Upgrade!

 

I, for one, would prefer the delay of a Release if a bug has been found by a fellow tester that affects some, but not all, instances of SIMS. After all, we might be one of the schools affected next time!

 

I agree with the approch taken by Capita to delay a release when a problem is discovered. This main issue I have is with the number of times that this is happening!!! I can't remember the last time that Capita actually released a product on time.

 

The worry I have is with the clear difference in the testing done prior to the UAT. Are their own internal processes actually that basic? I often undertake Beta testing on products so am not simply complaining for the sake of it

[Sivadam]

Teejay,

 

You can get the release made available for you to download on to a test machine. Or you can send the dataset down to Capita and attend one of their UAT sessions in Bedford. I do the latter.

 

Your dataset is loaded on to test machines and then upgraded to the new release. You are presented with scenarios of the new procedures in SIMS and test to see if they work as they should. It is also important that you test the other parts of SIMS that the new functionallity could affect.

A Feedback Spreadsheet is provided in which you log any issues that you have found.

 

The results of the testing are analysed to see if the release is fit for purpose.

 

At the last testing I spotted that the Send message, from a pupil to 'Teachers of Pupil', was only sending to the Pupil's Teachers for that day. This did not prevent release of the Upgrade but did go on record as something to be looked at for next time. I have referred to this in another thread, which is on SupportNet.

 

I have found this UAT procedure invaluable in getting ready for a new Upgrade. I have had an insight into new functionality and can inform our staff in advance of such new procedures, so that they are ready to utilise any innovations as soon as they are available.

An obvious & recent example was the NON-EDI Results in GCSE Exams (August release) - absolutely brilliant! We had our exact Headlines figures seconds after the Results were uploaded, because all extra results were already on the system!

Edited December 11, 2009 by Sivadam

[Sivadam]

I agree with the approch taken by Capita to delay a release when a problem is discovered. This main issue I have is with the number of times that this is happening!!! I can't remember the last time that Capita actually released a product on time.

 

The worry I have is with the clear difference in the testing done prior to the UAT. Are their own internal processes actually that basic? I often undertake Beta testing on products so am not simply complaining for the sake of it

 

I am sure that the Capita staff share your misgivings. I believe that they do have different datasets that they test on prior to giving it to us guinea pigs at UAT.

All I can say is that things have improved greatly since the move from db to sql happened - when I was in the forefront of those complaining.

But the problem is that different schools want different things from SIMS and we all use it in different ways. If several 'real' schools' datasets do not reveal problems, and then suddenly one does, how do you mitigate for such a scenario?

Edited December 11, 2009 by Sivadam

[Mcshammer_dj]

So do you get any form of benefit for this work that you do? I would guess that if the licence cost was lower for being in the UAT process then it may have certain benefits. Given that I have 70 databases to support I really do not have a great deal of time to spend looking at the UAT process.

 

We instead expect that the cost of the SLA allows for capita to deliver the product that we require, and also expect that the timescales for that delivery be met. Lets be honest some products (SM7) have consistantly been removed from the process meaning that some of the ways that the product works are still akin the the common platform days.

[Mcshammer_dj]

Equally it is worth noting that even once a product is finally released then we all test it again and often find problems are still present leading to the patchwork quilt that follows a release

[PhilNeal]

I'm afraid that you are expecting more than we can achieve and when faults are found we have to delay. Perhaps it is possible to do better and the team will try to do so.

 

We work 29 different returns into our releases that are defined at different times of the year and sometimes require significant clarification. We have to weave that into fixes and enhancements and deal with operating system changes. The delay is actually around 10 working days.

 

The returns for January were issued in October so as far as I am aware there isn't a pressing need to roll this update out to all schools.

Edited December 11, 2009 by PhilNeal

[Mcshammer_dj]

I'm afraid that you are expecting more than we can achieve and when faults are found we have to delay. Perhaps it is possible to do better and the team will try to do so.

 

We work 29 different returns into our releases that are defined at different times of the year and sometimes require signiicant clarification. We have to weave that into fixes and ehnacements and deal with operating system changes. The delay is actually around 10 working days.

 

The returns for January were issued in October so as far as I am aware there isn't a pressing need to roll this update out to all schools.

hey Phil I know that you all work hard to continue to deliver a service and we generally do appriciate all you do...

 

It just seems so disheartening that the move to seasonal releases to try to make it easier for us to deliver releases to schools without seeming to be months late has mean that we are again in the position that we tell schools that we are delivering the autumn 2009 update in 2010!!

[Mcshammer_dj]

The returns for January were issued in October so as far as I am aware there isn't a pressing need to roll this update out to all schools.

 

Not always the case as often when we find a problem we will be advised to apply the update as that may fix the problem!!

Edited December 11, 2009 by Mcshammer_dj
I can't spell

[teejay]

@Sivadam RE UAT testing:

Do Capita pay your school for doing this? It's just that spending time plus travel doing something that should really be included in the support/maintenance charges, our HT would expect to be reimbursed for my time and expenses.

[Sivadam]

@Sivadam RE UAT testing:

Do Capita pay your school for doing this? It's just that spending time plus travel doing something that should really be included in the support/maintenance charges, our HT would expect to be reimbursed for my time and expenses.

 

No they do not.

 

The only benefit that we/I gain, is an increased knowledge of the product and advance sight of what is coming, so that the school is in a better position than most to take up and use the new functionality. The school is well aware that we are highly regarded in our use of the SIMS Software mainly as a result of the breadth and depth of my knowledge. We do not have the issues that some schools may have of nobody having an thorough working knowledge of all the funtionality in SIMS. In effect, I use UAT in advance of an Upgrade in a similar way that others would use Training Courses for the new functionality after the release of an Upgrade.

 

I would also like to think that my opinion, on the way some of the new SIMS functionality develops, does hold some sway - even if this is only to a small degree.

Edited December 11, 2009 by Sivadam

[vikpaw]

@Phil - If it's legal, would it help if schools gave you permission to retain their data when they send it in, so that you can use this for testing.

 

I know the changes we make over time lead to more issues so static data isn't as great as a full UAT test, but having a more diverse source of test data might be more beneficial.

 

I often find odd and obscure problems on our systems and inevitably this leads to the need to send my data in, which takes a long time from here. I then work alongside the support team, both working on virtual setups to fix the problems. I'm sure this must happen to others as well, and though we may not be able to spare the time to test the new upgrades i'm sure there must be schools out their that wouldn't mind you retaining there data so long as the necessary security precautions were taken.

Edited December 13, 2009 by vikpaw
typo

[PhilNeal]

@Vikpaw - we do have schools that allow us to use their data in test (having taken steps to remove personal references). It's a vital part of our release process to go through as many data sets as we can.

 

Thanks for the thought Vikpaw.

[GrumbleDook]

@Sivadam RE UAT testing:

Do Capita pay your school for doing this? It's just that spending time plus travel doing something that should really be included in the support/maintenance charges, our HT would expect to be reimbursed for my time and expenses.

 

Were you a beta tester for windows 7? Are you on the office 2010 testing release? Did you *pay* for the beta of OS X? Did Google pay for you to test their email system for what seemed to be ages?

 

Beta testing is sometimes a labour of love and done for the betterment of the software involved, as well as allowing you time to prepare for it.

 

I know a few schools who do UAT with Capita and schools can opt not to do it for a particular release due to local reasons to the school. That seems to work pretty well.

[Jeremy69]

My school did UAT testing for a couple of modules and in each case we got to try it out and had some feedback into how the final product looked. Capita were appreciative. listened and did their best to sort any problems. Whilst the modules were in test the school got enhanced support so any problems could be quickly fixed. I might not have been paid but then I didn't expect to be and found the experience, on the whole, a rewarding one well worth doing.

[teejay]

Were you a beta tester for windows 7? Are you on the office 2010 testing release? Did you *pay* for the beta of OS X? Did Google pay for you to test their email system for what seemed to be ages?

 

Beta testing is sometimes a labour of love and done for the betterment of the software involved, as well as allowing you time to prepare for it.

 

I know a few schools who do UAT with Capita and schools can opt not to do it for a particular release due to local reasons to the school. That seems to work pretty well.

 

Neither do we pay any of the above examples thousands of pounds a year in support/maintenance fees;)

 

Try asking Capita to come to your school to test a bit of software for you, you'll get a nice hefty consultancy charge.

[Sivadam]

Jeremy,

This sounds more like a description of the Field Trial process. Enhanced support is obviously vital when you are participating in this programme, so that any major issues can be dealt with with utmost expediency. We/I also contribute to this process as well as to UAT.

A Field Trial is undertaken on a live system whereas UAT is on a test system.

Edited December 13, 2009 by Sivadam

[synaesthesia]

As anyone in the world of programising knows, it's *impossible* to release a 100% perfect product regardless of what tests you've run, how much data you have and how many people have beta tested it. In most circumstances the biggest glaring issues won't crop up and therefore be resolved until after it's been released to the general public.

[GREED]

Perhaps everyone is missing the most important point:

 

18 months ago Capita released upgrades on time, and then had big isues every time (I think we all remember the 4 releases of Feb 2008!).

 

Now they have moved to 3 main releases and are prepaired to test test test and if no joy put it back for more testing, we have very little major issues.

 

There will ALWAYS be programming problems, the fewer bigger ones the better. I am delighted that Capita take this approach and they can put it back weeks if it means we get a stable release.

 

OK it is annoying when we are waiting for fixes (and that is a gripe Phil that some fixes have to wait for releases rather than patches, but not your fault!), but better than it breaking everything on upgrade.

 

Many times there are 100s of changes made, I know from seeing at UATs, and each has to test how it affects the whole of the reat of the system.

[apeo]

Yeah I much rather put up with the Delays then get a release riddled with bugs. There was a time when i really dreaded upgrading as soon as the release was available (i delayed all my upgrades) but now not so much.

[garethEds]

Fair play to those who do put their data on the line for testing. Our SIMS is LEA controlled so we get updates when they say so.

 

I used to do a lot of testing for RM back when we were a CC2 and CC3 school. As previously mentioned it was a labour of love. I enjoyed testing the new stuff and had a group of pupils often working with me looking into the software they were sending.

 

Of course it sometimes went wrong - CC3 Service Release 4 was one major mess on a Friday afternoon - especially when their engineers all went home at 5. From that RM changed their times and if I remember correctly engineers were available later on Fridays - often when schools installed their updates. SR5 and 6 were smooth.

 

I remember the IE7 RM upgrade which developed problems with custom GPO settings.

 

One of the main reasons we did Field trials was access to the top line engineers and often they were willing to help you with other things that would usually go through first/second line support.

 

So well done those who do test SIMS. It cannot be easy.

 

GJE