The Analysis of Assessment Data in your school

[Guest MikeBostock]

For data managers and others who will wish to use 'raw' data in flexible ways, the need will remain for simple methods to interchange data in its raw state. So the need for an 'Export as...' button where there is data in a system that clearly ought to be able to come out, is not much to ask.

 

SIF will over time provide the standard method for moving data securely between large external databases, and different data systems that work across a school. Ideally, data systems should snap together like Lego and provide front ends that allow one to work at a conceptual level rather than have to think about the nuts and bolts of how the system works.

 

We haven't reached this point yet so flexibility is needed. Data managers will still need low-level data handling skills as well as high-level information skills. Over time they may need to drop the former as systems become easier to use. Its a bit like feeling that we shouldn't need to be a mechanic in order to drive a car. Performance Information Systems haven't yet got to the same evolutionary stage as the motor car, so we will still need to pack a few spanners in the toolbox.

[DBMan]

Mike, I have been reading all about 4Matrix as a result of your posts. Most impressive.

 

I agree with your point about not wanting data to leave schools. Can you shed any light on the advice you say was given by Becta (RIP) on this issue (FAQ page):

 

..."We do not as a rule accept data from schools because this goes against advice given to schools by by Becta"

 

Thanks!

[Guest MikeBostock]

Mike, I have been reading all about 4Matrix as a result of your posts. Most impressive.

 

I agree with your point about not wanting data to leave schools. Can you shed any light on the advice you say was given by Becta (RIP) on this issue (FAQ page):

 

..."We do not as a rule accept data from schools because this goes against advice given to schools by by Becta"

 

Thanks!

 

Dear DBman

 

Many thanks for the kind words.

The 'blog' has been my attempt to try to look at possible trends in how schools can improve their use of data systems in relation to school self evaluation and school inspection. I have been somewhat surprised by the number of visits that the blog has had, and the number of emails that I have received on these issues. (I should say that not all of them go along with my ideas - and I have learnt from these.) The 4Matrix application is the result of 8 years of working with schools on ways to add value to the data that schools already have. It is basically an SQL application that schools install and which tries to simplify the task of importing school data and processing it in the ways that schools have asked us for. Most of the features in 4Matrix are the result of responding to requests from data managers and assistant headteachers.

 

You ask an important question about sensitive pupil data leaving the school. I am not an expert on this but we have been strongly guided by the 'Data Handling and Security Guidance for Schools' from Becta. The link for this is here.

My concern is that it poses some challenges for both the online data services that schools find useful, and ideas like Cloud Computing in respect to school management information systems. The answer of course is to adopt acceptable levels of security. In our own case we took the view that the future of Performance Information Systems didn't lie in sending your data away like photos to be processed, but in having flexible tools that add to those that are already covered by a school's security policy. Our view is that information systems should become less 'fragile', more functional, and more available for every member of a school to use.

 

We have also ensured that our approach can be tried out without obligation by schools interested in adding to the tools that are already available to them. It is not an expensive system. I don't think there is one right answer to what is needed, but I am totally convinced that in years to come we will be getting much more value from the data available to us in Education, and that those school professionals who are involved with using data will find their roles much more about working with the school community on quality issues, and much less about abstract number crunching.

Edited August 17, 2010 by MikeBostock

[Martin_W]

I wanted to comment on the fact that some say that an MIS should be able to provide all the features needed to analyze assessment data.

 

From my experience in the dutch secondary education, I've seen that the MISs in Holland provide pretty similar features to those in England. However, the dutch systems all lack proper analysis tools. The hammer and saw analogy that Mike uses is quite appropriate. Most SIMs are created to be an administration system, not an multi-data source analysis system. To get to the data you need, without spending hours and hours of Excel craftmanship, is not always as easy as it might seem. And the usage of custom Excel sheets (from very small to supersheets (even pink ones ) is much more common than durable solutions that are built for proper data analysis.

[Guest MikeBostock]

From my experience in the dutch secondary education, I've seen that the MISs in Holland provide pretty similar features to those in England. However, the dutch systems all lack proper analysis tools. The hammer and saw analogy that Mike uses is quite appropriate. Most SIMs are created to be an administration system, not an multi-data source analysis system. To get to the data you need, without spending hours and hours of Excel craftmanship, is not always as easy as it might seem. And the usage of custom Excel sheets (from very small to supersheets (even pink ones ) is much more common than durable solutions that are built for proper data analysis.

 

Thanks Martin.

Let me give an example of the advantage in having extra tools in the toolbox....

 

This year, exam boards are not providing summary reports for each pupil.

This means that schools need to find a way to print the grade lists for every student.

Colleges will need this information on school headed notepaper.

 

This might be easy for some schools to do, but we were asked to provide a solution to this recently by a data manager who said it would take them for ever to do, and we have now added a new 'Student Reports' option which will print out summary reports in batches for every class onto headed notepaper. It is also a zero cost option for schools under free trial arrangements.

[Guest MikeBostock]

We have now added a new 'Student Reports' option which will print out summary reports in batches for every class onto headed notepaper. It is also a zero cost option for schools under free trial arrangements.

 

To illustrate what I mean there are pictures of the new reporting features ('Press Release' and 'Student Summary Reports') here. We reckon that schools would be able to do all their reporting in around an hour using additional tools like this. That includes importing the data from an MIS or spreadsheets.

 

I dont think you can ever stop looking for great additional tools for the professional toolbox. You have to weigh their value by the time they will save.

[workteam123]

Hi Tyiell

 

Is it possible I too can have a copy of this brilliant piece of work! Many thanks Gary

[pcstru]

Ideally, data systems should snap together like Lego and provide front ends that allow one to work at a conceptual level rather than have to think about the nuts and bolts of how the system works.

We haven't reached this point yet so flexibility is needed. Data managers will still need low-level data handling skills as well as high-level information skills. Over time they may need to drop the former as systems become easier to use. Its a bit like feeling that we shouldn't need to be a mechanic in order to drive a car. Performance Information Systems haven't yet got to the same evolutionary stage as the motor car, so we will still need to pack a few spanners in the toolbox.

 

I'm not quite sure who that is aimed at - the driver of the car in the case of most people in here I would assume to be the head. We ARE the mechanics. The head says that this year we need to make the car faster/make it better handling/get more MPG. We are part of making that happen.

 

MIS systems will never be able to provide all the answers and even tool-sets which can be coupled together using powerful interoperability standards will always be behind the curve when it comes to political policy and strategic goal setting in a specific school. The effectiveness of 'Data Managers' will therefore always be a function of their low level knowledge of the datasets and their ability to turn that into useful, up-to-date, accurate, easily digestible information. Interoperability might be nice, but the real glue that holds systems together is people and the real problem experienced by users of MIS systems today is transparent, flexible, real time access to data. MIS suppliers need to wake up to the fact that it is OUR data.

[Tyiell]

Hi Tyiell

 

Is it possible I too can have a copy of this brilliant piece of work! Many thanks Gary

 

Hi Gary,

 

Sure, I'll get her to put a sample together for you. PM me your email address and I'll get it sent over next week. It's not exactly a solution to the problem discussed here though, when all is said and done it is just a big spreadsheet. We basically create a series of large marksheets with if-then-else formulas to do some conditional formatting. Then we export it to excel and she organises in various sheets and runs formulas on them to pull out percentages, etc.

[Guest MikeBostock]

I'm not quite sure who that is aimed at - the driver of the car in the case of most people in here I would assume to be the head. We ARE the mechanics. The head says that this year we need to make the car faster/make it better handling/get more MPG. We are part of making that happen.

 

MIS systems will never be able to provide all the answers and even tool-sets which can be coupled together using powerful interoperability standards will always be behind the curve when it comes to political policy and strategic goal setting in a specific school. The effectiveness of 'Data Managers' will therefore always be a function of their low level knowledge of the datasets and their ability to turn that into useful, up-to-date, accurate, easily digestible information. Interoperability might be nice, but the real glue that holds systems together is people and the real problem experienced by users of MIS systems today is transparent, flexible, real time access to data. MIS suppliers need to wake up to the fact that it is OUR data.

 

Thanks pcstru for these observations. I agree that data managers need the skills that you describe, particularly providing teachers with high level information derived from low level data. One data manager that I am working at the moment in a Liverpool school is studying a Research Masters in data methods. She works directly to SLT and supports the 'research' activities that teachers are undertaking in the school into teaching and learning. Hers is a key position and a high level management role. That should be the direction that data managers' roles are developing, made possible by new data tools.

 

Vikpaw reminds me that I can't mention the software system that we produce but this school is using it to support this work. It does just what you say, i.e. provide up-to-date, accurate and easily digestible information. The raw data that it uses is the information taken from an MIS, and it will do this from SIMS, CMIS, Integris and Phoenix.

 

Once performance data is available in a flexible system like ours it can be processed and re-exported with ease, i.e. it is a flexible, universal performance toolkit. This system is not 'behind the curve' as you say, but leading the way in how data will be used in the future.

Edited October 15, 2010 by MikeBostock

[pcstru]

Thanks pcstru for these observations. I agree that data managers need the skills that you describe, particularly providing teachers with high level information derived from low level data. One data manager that I am working at the moment in a Liverpool school is studying a Research Masters in data methods. She works directly to SLT and supports the 'research' activities that teachers are undertaking in the school into teaching and learning. Hers is a key position and a high level management role. That should be the direction that data managers' roles are developing, made possible by new data tools.

I think the language is misleading. It is not (simply) made possible by new data tools, it is necessitated by policy (raising attainment) and made possible by people who know the the data (it's scope, limits, special meanings, constraints, quality), know how to access the data (whether that be through a raw database connection, a reporting engine or a third party bolt on) and who know how to manipulate the data into meaningful, digestible information which provide accurate metrics of the areas where policy needs to measure effectiveness to properly progress.

 

Once performance data is available in a flexible system like ours it can be processed and re-exported with ease, i.e. it is a flexible, universal performance toolkit. This system is not 'behind the curve' as you say, but leading the way in how data will be used in the future.

 

I would be amazed if this were true. I'd suggest it is not entirely true because in schools you have put it in, the tool is then configured to the needs of the school by people with expertise - and that is what brings it up to scratch in that particular school. It's not just software you sell surely, it is software, support and consultancy. Certainly when I read your descriptions of 'success stories', your intimate knowledge of what's going on in the school and the people suggest you are not just parachuting in a piece of software from afar and letting them get on with it? Which brings us back to the most important asset - the people. I can have a bucket load of the most sophisticated tools in the business but they are all useless without someone who knows how to drive them. Parachuting in another tool will do nothing unless the people learn how to use it and learn how it integrates with the various systems they have. Most people have the tools they would need in their boxes already - the MIS, the database(s), connectivity and a copy of excel. What they lack is the knowledge that allows them to make full and effective use of them.

[Guest MikeBostock]

I couldn't agree more with what you say pcstru. I am working with quite a few schools at the moment and that work usually starts with helping the data manager to get the best from the application, but then leads to working with senior managers to promote the use of the software (it's SQL and networked) and to help subject leaders to investigate patterns of achievement in their subject.

 

We provide a complete in-school support pack derived from the NCSL/TDA work on investigating in-school variation that starts with a checklist, then a presentation derived from the NCSL/TDA work on investigating in-school variation, and other materials designed to engage subject leaders in action-research.

 

It doesn't replace the list of tools that you mention, of which Excel is the most frequently-used in our experience, but provides a new toolkit that integrates and exchanges data with them. I have already argued that there will never be a point where a professional can sensibly say "I have all the tools I will ever need'. Please ask a school where it is being used whether they would now want to do without it. There is a link here to what some heads, DHs, AHTs and data managers have said about their experience of using it.

 

I am aware that quite a few members of this forum have already asked for a free trial of the system. Very few schools that have tried it have not gone on to adopt it, probably because it really does make more powerful use of performance data than is possible at the moment, and also that it is inexpensive and comes with free support for its use, including free remote tutorials for staff.

Edited October 16, 2010 by MikeBostock

[feltonam]

Hi Tyiell, I would find a copy of her spreadsheet useful too if it is possible for you to send me a copy?

[jenny_urey]

.............................