The Analysis of Assessment Data in your school

[PhilNeal]

As several have said, they've not really looked at PA which is where it all is!!

[garrysaddington]

As several have said, they've not really looked at PA which is where it all is!!

 

Extra purchase?

[vikpaw]

Extra purchase?

 

It's part of the assessment suite along with AM7, is that an extra purchase? I can't remember.

[PhilNeal]

Assessment & Performance Analysis are licenced as one product.

[CaterinaAnna]

If that is the case why is there so much discussion on here about using spreadsheets.

 

Don't know if my experience is typical but we have not used AM so far because:

- the last person who came close to understanding it was trained on an ancient version, reckoned (correctly at the time) that they could do much better using Excel, hasn't looked at it since and just happens to be in charge of these things;

- most teachers had no access to SIMS until last week since until then the firewall between our admin network and curriculum network was inviolate (and I don't mean painted purple);

- there's a stack of background and historical data to be imported if we're to make the best use of it;

- we got sidetracked for a while with e-markbook: great for teachers, hopeless for anyone wanting to work at the back end (sorry Kozan et al)

- and anyway I'm pretty comfortable doing it in Excel myself ;o)

 

I'm also pretty glad we didn't try using it too soon as it's only over the last year or so that we feel we're collecting only what we need, when we need it, in the most useful form so time could have been wasting setting up sets for data we never want to use again.

 

However a database is far better than a collection of spreadsheets and I see no point in designing one of my own when AM7's sitting there to use. So, once I'm certain SIMS is going to remain the school and LEA MIS of choice, then it'll be worth setting up all those aspects and result sets and using PA to get at least some of the pretty pictures and complicated tables made.

[DBMan]

It is unlikely that one company can provide everything that the market demands, so it is important that data can be made to move easily between applications.

 

True, but schools are held back by the fact that the data captured in 'MIS' systems such as SIMS is not as readily accessible as it should be, due largely to the 'off-limits' schema, nor is there a suitable method of accesing the data via pre-canned or custom reports delivered via an easy-to-use interface. If these issues were resolved the need to pay again for all of the bolt-on capability offered elsewhere would rapidly evaporate.

 

Imagine if Microsoft tried to exercise a monopoly and force people to only ever use text in Microsoft Word. There is so much that can be done with text that it is obvious that we must be able to cut and paste it into as many applications as covers the range of functions that we need. We are starting to think more creatively about performance data and this will stimulate a healthy desire for further data tools.

 

That's exactly what happens with SIMS...it captures the data but won't let you at it in any meaningful way, hence the bolt-ons that have arisen.

 

How may different ways can schools really all want to see performance data reported??? It should be delivered via the 'MIS' applications, then they would truly deserve the categorisation 'MIS'!

[DBMan]

I'd rather have a reliable, well-structured database that will produce the most commonly needed analyses easily, has a user-friendly front end and proper access to all the data than something that tries to be all things to all men & ends up frustrating people because it can't do things they're used to being able to do in excel or word.

 

Common analyses backed up by proper data access. Very well said and hardly asking for the world on a plate!

[Guest MikeBostock]

True, but schools are held back by the fact that the data captured in 'MIS' systems such as SIMS is not as readily accessible as it should be, due largely to the 'off-limits' schema, nor is there a suitable method of accesing the data via pre-canned or custom reports delivered via an easy-to-use interface. If these issues were resolved the need to pay again for all of the bolt-on capability offered elsewhere would rapidly evaporate.

That's exactly what happens with SIMS...it captures the data but won't let you at it in any meaningful way, hence the bolt-ons that have arisen.

How may different ways can schools really all want to see performance data reported??? It should be delivered via the 'MIS' applications, then they would truly deserve the categorisation 'MIS'!

 

It is about choice and availability at any particular time. No serious specialist would go about their work with tools missing from their toolbox. They don't say "I will only tackle jobs that need a hammer and a saw". Neither do they argue that the toolbox they bought should have had all the tools that they will ever need in it, and begrudge buying further tools.

 

The way that performance data is currently used in education is actually relatively primitive. We only have to ask if we will still be accepting today's constraints a few years from now to appreciate this. There is huge scope to make a giant leap forward. I have written about this subject in my blog 'Wrestling with the Data Genie'. MIS suppliers are doing their part to provide greater functionality and interoperability. Smaller companies like my own are also helping to provide affordable solutions to the growing demands on schools to make their data work harder.

Edited May 19, 2010 by MikeBostock

[Dr_bob]

IMO spreadsheets are not an elegant solution. They may allow you to analyse your data but then ytou have the problem of the data being in multiple places which can cloud its accuarcy (you either have to have a link to the back end databse or risk not knowing of data has been updated since you performed your extract). Its much better to have the reports as an option from the source database imo with as little end user input as necesary required. This mitigates the risk of user errors and ensures that you have consistancy in the data being used for any analysis.

 

Not saying spreadsheets are not useful - far from it - but if the data you are using in them is inacurate (for any reason) then the analysis you are performing is useless.

Edited May 25, 2010 by Dr_bob

[vikpaw]

you're right dr_bob, but we have to do what works, and in my case, we would get all the data in termly, it would be chased up if missing, and sent home as reports. we then exported to excel to analyse and reimported certain calculations as necessary. the data could change, but was unlikely to.

[LosOjos]

you're right dr_bob, but we have to do what works, and in my case, we would get all the data in termly, it would be chased up if missing, and sent home as reports. we then exported to excel to analyse and reimported certain calculations as necessary. the data could change, but was unlikely to.

 

Alternatively, if you're using SIMS (I'm sure something similair could be done in other systems but I've never used any of them), you can set up Access to extract data from SIMS via CommandReporter to get an "almost live" reporting system. Hopefully SIF will make this whole process simpler. I'm surprised that a company the size of Capita that has a system the size of SIMS hasn't looked at buying up a third party reporting system (which seem to be getting more and more popular and hence costly) and incorporating it in to the Assessment module, as clearly the current analysis tools just do not cut it for most people...

[vikpaw]

Alternatively, if you're using SIMS (I'm sure something similair could be done in other systems but I've never used any of them), you can set up Access to extract data from SIMS via CommandReporter to get an "almost live" reporting system. Hopefully SIF will make this whole process simpler. I'm surprised that a company the size of Capita that has a system the size of SIMS hasn't looked at buying up a third party reporting system (which seem to be getting more and more popular and hence costly) and incorporating it in to the Assessment module, as clearly the current analysis tools just do not cut it for most people...

Have you tried the same with an SQL db? i'd like to give that a try.

It could be an exact replica of SIMS just with our additions on collating data, and reporting. Then we could offer it to other schools as an add-on, and maybe Capita would buy us out...

[LosOjos]

Have you tried the same with an SQL db? i'd like to give that a try.

It could be an exact replica of SIMS just with our additions on collating data, and reporting. Then we could offer it to other schools as an add-on, and maybe Capita would buy us out...

 

With an SQL db (which let's not forget SIMS is!) it would be even simpler, you could even have the entrie system available online and therefore platform independent and available anywhere! Unfortunately, we're not allowed to mess with the SIMS db, and even if we were, it changes with every update. I'm sure Capita have their reasons for doing things this way, but the cynic in me can't see past the dependence we are left with to pay for modules that don't perform the tasks we need (or at least not in a way that anybody can be bothered to use them!)

 

On a positive note, SIMS isn't all bad (as much as we might complain), it just needs to be a bit more user friendly in the assessment and analysis modules, or open up the DB so that those with the knowledge can create custom reporting solutions that fit their needs and always contain the latest available data

Edited May 25, 2010 by LosOjos
On a positive note...

[Dr_bob]

On a positive note, SIMS isn't all bad (as much as we might complain), it just needs to be a bit more user friendly in the assessment and analysis modules, or open up the DB so that those with the knowledge can create custom reporting solutions that fit their needs and always contain the latest available data

 

this for me is the nub of it all. Sims, in itself, isnt a bad product but I do get the feeling that Capita want to milk it for all its worth with the sale of add ins\ons and additional modules. The ability to make accurate decisions based upon data relating to education and what after all is the future of the country isnt one that IMO should be restricted to thsoe institutions who have enough money to pay to get access to their own data. Education shouldnt be about commercial interests and profit but about enabling schools to make decisions that will enhance their capability to provide the best education they can.

 

and now back on reality beach ....

[Guest MikeBostock]

The over-riding aim here is to improve the data tools that schools use so that data professionals and IT managers in schools can be more effective in the way that they use performance data.

 

I have suggested previously that schools need a number of data tools, and easy ways to move data about. I have made an analogy with Microsoft Word, which has a vast choice of useful features, but of course, people will also use Word in parallel with other applications.

 

But using data in a similar way to text is much more difficult. This is partly because of the need to preserve the relationships and interdependence of data held in many separate tables. It isn't really just a question of writing the correct SQL query, because it isn't enough to just pull data from one place and put it in another. What is more important than data is meaning. If one wishes to add greater meaning to data then this can be done by combining data from different sources and performing new mathematical operations on the combination. This is an example of where it will usually be more efficient to acquire a new tool that does just this. We could make a hammer, but it will usually be more sensible to buy one.

 

In our work, we have aimed to provide data tools that will selectively aggregate data from different sources, and other tools that let the user manipulate it in ways that can't easily be done in Excel. It then uses this information in conjunction with a live database of every UK examination, and another containing national comparative subject performance data. The result is that new information is revealed, and this can then be exported in different formats to be reused yet again.

 

I believe that it is very good for schools to have different companies providing solutions, new features and choices for schools. No single, monolithic approach will work for everyone.

[PhilNeal]

For those that are interested, there is a web cast on 16th June @ 1.30 "Spotting the trends – Can this improve school performance?" Details on SupportNet.

[Guest MikeBostock]

A follow up to this thread. I was looking for a CMIS thread on the subject because of the availability now of equivalent routines for using the examination, pupil and teaching group information from both CMIS and SIMS. Many have looked to the Schools Interoperability Framework for news of easy movement of data between various tools, including Excel, that data professionals will wish to have in their toolbox. With the forthcoming closure of Becta it is uncertain how this work will continue.

 

We have always taken the view that data exists in native forms anyway (XML, CSV, XLS, TXT, DOC etc) and that simple compatible export and input routines on every tool in the toolbox would provide the interoperability that people need. Our developments permit maximum data movement by these routes, coupled with active extraction where any software application does not provide export options.

 

From August 24th schools systems will be tested with the arrival of the new examination results. This will be the time to check whether there are sufficient tools in the box to meet the new reporting requirements from LAs that include discounting, disapplication, capped and uncapped results and separate reporting for science and modern languages. If the processing of such reports requires the use of Excel and the possibility of the usual range of common processing errors, it will point to a possible need for further development by schools of the range of tools that they are using.

 

The point of this note is simply to argue for the benefit of reviewing one's IT systems at mission critical times, and to point to the value of simple interoperability protocols for moving data between the tools that can perform the tasks that are are needed, in the timeframe that is available, so as to create maximum impact.

 

The popular blog at Wrestling with the Data Genie describes many of the issues associated with getting better value from school data systems. It is an area to keep an eye on because of the potential for innovation. Both MIS suppliers and others are working on developments to make life easier for data professionals in schools.

Edited July 25, 2010 by MikeBostock

[chapman23]

... Many have looked to the Schools Interoperability Framework for news of easy movement of data between various tools, including Excel, that data professionals will wish to have in their toolbox. With the forthcoming closure of Becta it is uncertain how this work will continue...

 

Just to reassure everyone, despite the forthcoming closure of Becta, the Systems Interoperability Framework will continue, as the SIF Association is a non-profit membership organisation. There is a press release explaining this in the latest SIF Association UK newsletter:

 

The SIF Association UK, through the leadership of Becta, has become an efficient, widespread and self-sustaining organisation in the UK. While the new UK government announced the closure of Becta, the SIF Association UK will continue with the community membership continuing to manage the critical work ahead. The elected Management Board of the SIF Association UK continues to drive the vision and strategy of interoperability in the UK.

[chapman23]

Just to reassure everyone, despite the forthcoming closure of Becta, the Systems Interoperability Framework will continue, as the SIF Association is a non-profit membership organisation. There is a press release explaining this in the latest SIF Association UK newsletter:

 

The SIF Association UK newsletter can be found at http://www.sifinfo.org/uk/upload/press/B6E2ZD_SIFAssociationUK_Newsletter_Issue_14_June2010.pdf

[jamesleeds]

 

Dos_Box

Edited October 14, 2010 by Dos_Box

[Guest MikeBostock]

Is SIF really the way forward? Zone servers etc seem to be unnecessarily complex at best, and a real constraint on interoperability (and waste of resources) at worst.

 

James raises an important question. SIF is really quite a complex and expensive data channel that isn't ready yet, when what we really want is a simple button that says 'Export data as...'

Does anyone remember a cartoon about a tree swing called 'What the Customer really wanted'?

 

History also shows that we can wait around for something promised to provide all the answers and then find that what we really needed was the step beyond that.

New opportunities to get better value from schools' data comes from using it together with other information from national datasets and from the school's particular context. Importing additional data is as important to this as exporting the data you already have. Sorting out interoperability of data is just a modest first step in making school data systems work harder.

 

In our project, developments are driven by what headteachers and data managers have asked us for. They have asked for easy ways to move their data between the tools they already have, ways to represent pupil performance that improve on grade lists and bar charts, tools to do the essential data-crunching for them without the usual embarrassments you get when working in Excel, and a way that makes it easy for teachers to use data without needing to be statisticians. We have also made sure that data professionals can try these ideas out for themselves instantly and without commitment.

 

ICT keeps moving forward and there are seldom single right answers. The world of performance data has much catching up to do, which is why being opportunistic and open minded is an asset.

[chapman23]

SIF can be as simple or as complicated as you want to make it. A Zone Integration Server can handle the asynchronous transfer of data between applications in a school or college, between institutions (for 14-19, or potentially to replace CTF) and for vertical interoperability from schools to LAs or the Department for Education (as a potential replacement for Census?).

 

With more and more suppliers joining the SIF Association and implementing SIF in their existing products, or using third party agents to ensure interoperability, we are not too far off a situation where heads and data managers can easily transfer data into their existing tools. A list of SIF products currently available in the UK can be found on the SIF Association website.

 

SIF is driven by its members, with new developments coming from teaching staff and data managers via a number of working groups. Version 1.3 of the SIF Specification is almost at the end of a 30 day consultation period (closing tomorrow!) so if anyone wants to provide any feedback, please email penny.murray at becta.org.uk

[matt40k]

But someone needs to pay for SIF, nothing is free. Ideally the "LA" would have a ZIS, and the schools would connect to it and the "LA" to the rest of the market, other LAs, DfE etc. SIF within the school across software sounds wonderful, but it's another service that schools need to support. Again, more cost.

 

The idea of Census being done via SIF sounds wonderful, it could be a more regular thing, however Census is an excellent time for data cleaning.

[chapman23]

But someone needs to pay for SIF, nothing is free.

 

Yes, but someone also has to pay for the work that goes into the bespoke "Export data as" button, or IMS Enterprise or numerous APIs. The benefit of using SIF is that the ZIS can be used to allow multiple applications (and institutions) to share data.

 

Take a look at this presentation from an assistant head at one of the schools involved in the SIF Association. He illustrates a number of the benefits of implementing SIF: reductions in bureaucracy, improving the quality of data, improving data security, freeing up time for staff to teach, improving child safety etc. He also estimates that his school could save 400 hours a year on data management by implementing SIF.

 

SIF can help improve data quality as you can enter data once into the MIS rather than having to retype the same info into every application and means you don't have to manually update every app when some info changes.

 

The idea of Census being done via SIF sounds wonderful, it could be a more regular thing, however Census is an excellent time for data cleaning.

 

Rather than relying on Census to cleanse data, it would be better to have processes in place to ensure data is up to date, relevant and accurate. Becta's Information Management Strategy Framework may be helpful in doing this :-)

[matt40k]

I agree, but the point still is, it's used as a time to cleanse data and the cost (in human hours) is embedded into the census process rather then, say, added more work to the actual day-to-day operation. I suppose you could integrated it into the "Ofsted" checks, but it's giving it another name and causing more work changing it.

 

Did you see Becta cost estimate of pushing out SIF? Personally, SIMS (school) can export into ONE (LA), it's been working for a number of years. I admit it's not perfect, but for most of the 22,000 schools, SIF won't give them much benefits. On paper it brilliant but in the real world of budget cuts...