The Analysis of Assessment Data in your school

[Kozan]

Hi all

 

I’m an analyst working for Bromcom Computers. I have been assigned a task to spec’ a new assessment analysis software module.

 

The primary objective of the software module is to give schools a powerful but easy to use software analysis tool that can quickly collate, filter, sort, plot/chart and tabulate recorded assessment data into information that schools can use to determine and track performance from the student level > subject level > department level > school level.

 

Now, I confess I am a complete newbie when it comes to analysis of assessment data and my present knowledge and understanding is limited. Therefore, I would appreciate any information and guidance on this topic. I am interested in both primary and secondary schools’ responses. Specifically:

 

1. What kind of assessment analysis do you presently do in your school?

 

For example do you produce: Student/Subject Residuals, % of students achieving certain grade ranges for the core subjects?

 

2. What kind of trend/predicted analysis do you do?

 

3. How do you make use of “Value added” data such as FFT, Yellis, MidYis, Raiseonline?

 

4. How much of this analysis is done in your MIS?

 

5. How much of the analysis done outside of your MIS – say in Excel?

 

6. How to you choose to present your data: Tables? Bar graphs? Line graphs? Colour coded conditional formatting?

 

7. How often is this analysis done?

 

8. Who does this analysis in your school?

 

9. What problems/difficulties do you encounter with your current analysis process?

 

10. What features and functions would you like to see in assessment analysis module?

 

Please feel free to add any other comments.

 

Thanks in advance.

[resin1]

Hopefully you will get a lot of input Kozan - I'd add more if i had the time, but as a quick response:

 

from a secondary school perspective, if the module you developed would support the type of usability and presentation being offered currently by external commercial solutions such as 4Matrix and Sisra Online, to review current progress as well as final results data, then you would win over school leadership teams no problem whatsoever!

 

By implication, at KS4 this means the assessment module should allow users to define means of recording grades against elements that have their (DCSF) performance measure contributions built in, so that the analysis part of the system can do all the hard work for the user, and allow the level of top level decision making for intervention etc that the above examples are currently providing.

 

I have always felt that the MIS providers should be delivering this now, but I'm not sure any of them get anywhere close - if you look at it from the point of view of the amount of work and skills required to set-up analysis and then the user interfaces provided to exploit the data.

 

Monitoring progress at KS4 is a good example I feel - it simply shouldn't be necessary to have to fashion a recording mechanism with an MIS, build in intelligence by using whatever formula mechanism to summarise key criteria by (by which schools are judged), and then fashion a whole host of different "views" of that data to provide a way for decisions to be made from it.

 

Catering for all possible types of qualifications that schools may be delivering is a big ask, but I don't see it as an unreasonable expectation from an MIS system, when there are MIS systems out there that support this level of "intelligent" analysis for summer exam results.

 

Hope that helps.

[Kozan]

Thank you Resin1 - most useful. I had been looking at SISRA online prior to my post - but I don't know much about 4matrix. I'll check it out.

 

Based on my research thus far, it seems to me that schools are looking for and would value a graphical/visual approach to data analysis - since most people receive 80% of thier information visually. Would you agree?

 

I do appriciate your time - but if you have another spare five minutes - please could you expound on the following statement:

 

"the assessment module should allow users to define means of recording grades against elements that have their (DCSF) performance measure contributions built in, so that the analysis part of the system can do all the hard work for the user, and allow the level of top level decision making for intervention etc that the above examples are currently providing."

 

Thanks in advance.

[dhicks]

What kind of assessment analysis do you presently do in your school?

 

Basic percentages-of-grades.

 

What kind of trend/predicted analysis do you do?

 

I forget the exact name and algorithm at the moment, but something provided by one of the members of staff (who is a good statistician and knows what she's doing).

 

How do you make use of “Value added” data such as FFT, Yellis, MidYis, Raiseonline?

 

I'm not sure.

 

How much of this analysis is done in your MIS?

 

None.

 

How much of the analysis done outside of your MIS – say in Excel?

 

The analysis is currently done by a set of Python scripts that takes in a set of raw figures and produces a bunch of tables and charts. This is flexible but not very maintainable (i.e. I know how it works, no-one else does). Unfortunatly, none of our teaching staff have enough knoweldge of Excel to be able to use it.

 

How to you choose to present your data: Tables? Bar graphs? Line graphs? Colour coded conditional formatting?

 

Tables, assorted charts.

 

How often is this analysis done?

 

Once a year, which much wailing and gnashing of teeth, although given more data updates the script could easily be run every night or however often is required.

 

Who does this analysis in your school?

 

Me.

 

What problems/difficulties do you encounter with your current analysis process?

 

Teaching and admin staff don't know enough about statistics to be able to properly articulate what it is they want to show, and if they did then they lack the IT skills to convert that articulation in to tables or charts. It should be perfectly possible to knock something together with Excel but they simply do not have the skills to do that.

 

What features and functions would you like to see in assessment analysis module?

 

It needs to produce "nice looking" charts that we can use for marketing purposes. It needs to be simple to use, with no fuss, for people with few basic IT skills.

 

--

David Hicks

Edited March 30, 2010 by dhicks

[resin1]

ok Kozan - quickly ;-)

 

yes - visual, visual visual

 

what I mean by that statement, is that I'd want to be able to set-up a progress tracking system where for example, I say for this subject we need to collect predicted grades that are full GCSE (A*, A, B, C etc) and I am able to chose an element that allows those types of grades to be input and within the system the element is classed as "1 whole GCSE" or in other words 20% contribution to Level 2 performance measure threshold (5+ A*-C) and that a A* is worth 58 QCA points. Similarly, I could assign a GCSE grade collection element for a GCSE short course, and the system would know it is 1/2 a GCSE/10% contribution/A*=29 etc. Or for an OCR National Level 2 Certificate, I could collect Pass/Merit/Distinction and it would know these are 4 GCSE/80% contribution/Distinction=220 points etc.

 

see Home Page - National Database of Accredited Qualifications if you are not familiar with all that stuff (search for a qualification and look at its performance measures).

 

Although I haven't used either of the those systems I mentioned, I believe what they provide is a mechanism to map subject name/raw grade data collected via MIS or other, to an internal qualification database that then means they can analyse that raw data, present it meaningfully and allow drill down by users for decision making, in a context within which those users are being judged by external parties.

 

Although I'm sure there will be a place in the market for them for some time to come and I'm not wishing them out of business (apologies if any of their reps look in here) but it would seem to me that an MIS should be able to offer something for assessment/progress monitoring that is designed and fit for the purpose(s) of a large number of users out there, rather than delivering something that is generic and may allow all to do something, but doesnt really fit a specific purpose.

[PhilNeal]

I'm sure all these companies will appreciate you all helping someone copy their ideas!

[vikpaw]

hi kozan

worth making the module talk easily with external systems, and have flexible input and output of data. e.g. raiseonline, and fft data, both heavily used by ofsted.

possibly worth contacting those bodies as well.

there's also another body to do with ks2/3 grades that i made contact with some time ago, i forget the name, i think it's the people that send the CAT scores out, worth talking to them, the developer was most helpful to me, once i'd socially engineered his private mobile that is!

good luck.

[vikpaw]

I'm sure all these companies will appreciate you all helping someone copy their ideas!

 

Can't harm to see what's on the market to see what you can Discover :troll:

[skunk]

I'm sure all these companies will appreciate you all helping someone copy their ideas!

 

Someone needs to pull of the ideas from these disparate systems and put them all in one system that schools can use effectivley, we cannot afford five systems that each do a little bit well. My MIS's analysis suite currently does not do half (or even a tenth if I am honest) the things I would wish, but I would be happy to swap if someone were to write one that did all the great stuff I have to do by hand in excel currently.

 

Students matter, products do not.

[MattMitchell]

I'm sure all these companies will appreciate you all helping someone copy their ideas!

 

That occurred to me too, Phil. Certainly, the one area where most MIS software is lacking is in providing ready-made analyses of the kind that schools actually want; marketing materials from the various vendors (Bromcom and Capita included!), alongside blanket questionnaires like this, suggest that this is unlikely to change in the immediate future.

 

I don't really want to write a full analysis suite spec for free, but I will give some tips:

 

• Have a read through the literature - there are many guides online to analysing assessment data available, which will give you an idea of typical foci of interest. The national strategies sites, forums such as this one, LA support sites, etc, have guidance on how to measure performance, and what to look for.
  
• Talk to some schools, i.e. go and visit them and see what they do - then you can think about how to solve the problems they're having to solve themselves at the moment
  
• Look through the feature requests for products like TeachersWebFolder - would any of these fit your brief?
  
• Have a read through DCSF and Ofsted guidance on how to assess the effectiveness of schools and LAs - this will give you an idea of the kind of questions schools will need to answer
  

 

It's going to be difficult to make anything easy for schools for as long as MIS software allows users to store assessment data the way they want. A lot of the time, schools tend to be very focussed on colour-coded lists of students: a quick search online will show you what they're asking about!

 

If this is a sanctioned project at Bromcom, you may want to look at getting in a few consultants (some who are "data-focussed" and some who are "anti-data") to discuss what they think would be helpful. Is this a Bromcom thing? I would imagine the firm is big enough to pay a team to develop a solution, and not just to let a new analyst "have a go" based on what they can find in a discussion forum...

 

Overall, the kind of questions you need to answer are:

• What's going wrong, and for which subject/students/groups of students?
  
• What's working, and (again) where is it working?
  
• How did we do in the past? How will we do in the future?
  
• Where do we need to improve in order to hit XYZ threshold target?

[resin1]

The above comments made me smile - especially Vik's - nice one.

 

Certainly dont see the need to defend my self, but I'll just say that any vendor worth their salt should be constantly reviewing and researching to see what other vendors or complimentary solutions are up to, in order to either stay one step ahead of the competition or try to at least deliver something that is competitive. Anything stated earlier could be gleaned within a few minutes perusal of their web-sites, which I'm sure, Kozan's marketing or product management colleagues would hit sooner or later if he didn't himself.

 

To me they were just examples of what users might demand. Heaven forbid that one vendor might be influenced by another commercial offering

[Matthews]

Hi

 

I am currently looking for some software that can produce this kind of analysis. I currently manually create analysis from Assessment Manager7 and spreadsheets. We are a large school and this can be an extremely time consuming process, especially now that we have to produce minority analysis along side the normal analysis.

 

We need software that can tell us:

 

Percentage A*-C in KS4 & 5 in each subject for Assessments, Targets & FFT or Alis

Percenatge 5+ & 6+ in Ks3 Subjects for Assessments, Targets & FFT

 

This data also needs further analysis to tell us how many of these are Girls,Boys,SEN,FSM,EAL & Ethnic Minorty Students.

 

Currently the process i'm using takes me two weeks to collate.

 

I'm sure i'm probably doing it in a much longer way than needed.

 

Can anyone tell me what software (if any) they use? Or how they use SIMs to generate such analysis that may cut my time spent doing our termly analysis.

 

Any help is appreciated.

 

Kelly

[Tyiell]

@Mathews: Our brilliant data manager has built a phenomenal spreadsheet that does this and more - it's mind-boggling to behold, but she can more or less export the data from sims and copy it in and boom - it all works. I can ask her if she will let me post it for you if you like, but be warned - it's MAAASSSSIVVE (and pink, she likes all her spreadsheets to be pink!!).

[Matthews]

lol... Tyiell

 

I have tried to do a spreadsheet to house analysis and i can get one to do KS3 as our students in KS3 all do the same subjects. It gets more complicated in KS4 due to their option choices etc...

 

But yes please if you could post it or send me a copy i'd be enternally grateful!!!

 

I would ofcourse want to pay her (in biccies) for her trouble

[kxv2020]

@Mathews - can't you use analysis and trend tracking tools in AM7?

[Tyiell]

@Mathews - can't you use analysis and trend tracking tools in AM7?

 

Don't know about Matthews, but we have tried those, and while they are useful, we find that they don't provide the flexibility we need to satisfy the powers that be!!! Wish it did!!!

Edited April 20, 2010 by Tyiell

[garrysaddington]

Hi

We need software that can tell us:

 

Percentage A*-C in KS4 & 5 in each subject for Assessments, Targets & FFT or Alis

Percenatge 5+ & 6+ in Ks3 Subjects for Assessments, Targets & FFT

 

This data also needs further analysis to tell us how many of these are Girls,Boys,SEN,FSM,EAL & Ethnic Minorty Students.

 

 

Kelly

 

Do you mean like your MIS supplier should have already been doing, without making schools do the work. See the link below (from 2008).

 

Management Information Systems (MIS) and pupil tracking: Developers' guidance

 

What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it. Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS.

[Tyiell]

That document seems to highlight what you should be doing without really going into how - most particularly it is not specific to Sims. I can't speak for CMIS, but SIMS does not currently have an in-built solution to provide the extensive data analysis we are asked to provide (although I believe we are asked to provide slightly more data than the guidelines require). It does have some tracking and assessment tools which we use quite effectively, but they are not complete, nor are they flexible enough to allow us to tailor the results the way we do in our tracking spreadsheets.

 

Obviously it would be preferrable to keep it in the mis, but until the features are available, we have no choice...

[garrysaddington]

That document seems to highlight what you should be doing...

 

Not what you should be doing, what your MIS supplier should be doing, there is a separate document for schools. This is the other doc.

 

Management Information Systems (MIS) and pupil tracking: Users' guidance

[LosOjos]

I think with SIMS in particular, the tools are all there to get the info you need, but where it falls flat is the very limited functions available in AM7, making it much easier in most cases to extract the lot to Excel and mess with it there.

 

If Capita improved the functions available I think AM7 would be up to the task, but they'd need to at least make if statements have a LOT more cases, 3 and an 'else' doesn't cover it, plus nesting functions: it's simply not practical to have 8 columns hidden away just to get to one result, it makes marksheets huge, messy and difficult to manage.

 

It'd also be nice if you could add whichever columns you wished to a marksheet from elsewhere in the system, the available 'additional student info' columns are extremely limited when you look at the vast amounts of data that are stored in SIMS, which must all be linked up to a student by some common identifier, so why can't we get at them?

 

One thing that has always bugged me too is how awkward it is to switch between result sets, you have to either go in to each column individually and change it, or delete the lot and add them back in with the new result set: if you could highlight a range of them and update the result set in one go, that'd speed things up a hell of a lot.

Edited April 21, 2010 by LosOjos

[Tyiell]

I think with SIMS in particular, the tools are all there to get the info you need, but where it falls flat is the very limited functions available in AM7, making it much easier in most cases to extract the lot to Excel and mess with it there.

 

If Capita improved the functions available I think AM7 would be up to the task, but they'd need to at least make if statements have a LOT more cases, 3 and an 'else' doesn't cover it, plus nesting functions: it's simply not practical to have 8 columns hidden away just to get to one result, it makes marksheets huge, messy and difficult to manage.

 

It'd also be nice if you could add whichever columns you wished to a marksheet from elsewhere in the system, the available 'additional student info' columns are extremely limited when you look at the vast amounts of data that are stored in SIMS, which must all be linked up to a student by some common identifier, so why can't we get at them?

 

One thing that has always bugged me too is how awkward it is to switch between result sets, you have to either go in to each column individually and change it, or delete the lot and add them back in with the new result set: if you could highlight a range of them and update the result set in one go, that'd speed things up a hell of a lot.

 

I agree with all of that - and could add a litttle more: Performance Management is getting better for pupil tracking in AM7, but there are still issues with filter groups: you can filter by SEN Status, but not by SEN Code. You can filter by EAL, but not by ethnicity. Not sure why they don't just let you filter by anything recorded - that is what databases are designed to do after all!!!

[Matthews]

Hi Kiran

 

There's only so much that AM7 can do (As far as i know)

We do have things set up that tells us how many students getting 5 A*-C etc... but subject by subect they want to know percentages getting 5A*-C and considering we do 15 to 20 subject depending on the year it all becomes a little laborious to right click above each column (aspect) to get those percenatges. If it's a case that i'm doing it the longest way then i'd appreciate your advice on the quickest....

 

Currently i'm right clicking on the headings and asking for grade distribution to find the percentages. I'm then having to do group filtering to find the percentages for each category (Fems,Males,SEN.FS,EAL & Eth) so as you can imagine this becomes very long winded. Each subject each year for each category. Ofcourse we do have things in place that tell us 3 levels progress ect... but i feel as though i'm having to look in several different places to compile one analysis report.

 

Any help is appreicated Kiran

[Matthews]

Do you mean like your MIS supplier should have already been doing, without making schools do the work. See the link below (from 2008).

 

Management Information Systems (MIS) and pupil tracking: Developers' guidance

 

What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it. Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS.

 

Hi garrysaddington

 

It doesn't half feel like you're completely on your own when it comes to manipulating SIMs to produce this information. I would say i'm quite good at using and manipulating SIMs or AM7, but this one has stumped me.

 

If i don't get this one sorted then i'm stuck doing a full two week painfully boring step by step day by day analysis, with mixture of extracting,exporting,manipulating and calculating tons of data.

 

Edited April 21, 2010 by Matthews

[Matthews]

That document seems to highlight what you should be doing without really going into how - most particularly it is not specific to Sims. I can't speak for CMIS, but SIMS does not currently have an in-built solution to provide the extensive data analysis we are asked to provide (although I believe we are asked to provide slightly more data than the guidelines require). It does have some tracking and assessment tools which we use quite effectively, but they are not complete, nor are they flexible enough to allow us to tailor the results the way we do in our tracking spreadsheets.

 

Obviously it would be preferrable to keep it in the mis, but until the features are available, we have no choice...

 

I totally agree!

[kxv2020]

Hi Mathews

 

In part I agree - it isn't the most friendly; but you can use aspect analysis which does help the with GCSE WAGs and Target Grades with filters for SEN etc. Have a play - that's all I do and see if it works. Apart from that we do much analysis in Excel as the LA use it for their tracking too!

Edited April 22, 2010 by kxv2020