The Analysis of Assessment Data in your school

Hi all

I’m an analyst working for Bromcom Computers. I have been assigned a task to spec’ a new assessment analysis software module.

The primary objective of the software module is to give schools a powerful but easy to use software analysis tool that can quickly collate, filter, sort, plot/chart and tabulate recorded assessment data into information that schools can use to determine and track performance from the student level > subject level > department level > school level.

Now, I confess I am a complete newbie when it comes to analysis of assessment data and my present knowledge and understanding is limited. Therefore, I would appreciate any information and guidance on this topic. I am interested in both primary and secondary schools’ responses. Specifically:

1. What kind of assessment analysis do you presently do in your school?

For example do you produce: Student/Subject Residuals, % of students achieving certain grade ranges for the core subjects?

2. What kind of trend/predicted analysis do you do?

3. How do you make use of “Value added” data such as FFT, Yellis, MidYis, Raiseonline?

4. How much of this analysis is done in your MIS?

5. How much of the analysis done outside of your MIS – say in Excel?

6. How to you choose to present your data: Tables? Bar graphs? Line graphs? Colour coded conditional formatting?

7. How often is this analysis done?

8. Who does this analysis in your school?

9. What problems/difficulties do you encounter with your current analysis process?

10. What features and functions would you like to see in assessment analysis module?

Please feel free to add any other comments.

Thanks in advance.

Hopefully you will get a lot of input Kozan - I'd add more if i had the time, but as a quick response:

from a secondary school perspective, if the module you developed would support the type of usability and presentation being offered currently by external commercial solutions such as 4Matrix and Sisra Online, to review current progress as well as final results data, then you would win over school leadership teams no problem whatsoever!

By implication, at KS4 this means the assessment module should allow users to define means of recording grades against elements that have their (DCSF) performance measure contributions built in, so that the analysis part of the system can do all the hard work for the user, and allow the level of top level decision making for intervention etc that the above examples are currently providing.

I have always felt that the MIS providers should be delivering this now, but I'm not sure any of them get anywhere close - if you look at it from the point of view of the amount of work and skills required to set-up analysis and then the user interfaces provided to exploit the data.

Monitoring progress at KS4 is a good example I feel - it simply shouldn't be necessary to have to fashion a recording mechanism with an MIS, build in intelligence by using whatever formula mechanism to summarise key criteria by (by which schools are judged), and then fashion a whole host of different "views" of that data to provide a way for decisions to be made from it.

Catering for all possible types of qualifications that schools may be delivering is a big ask, but I don't see it as an unreasonable expectation from an MIS system, when there are MIS systems out there that support this level of "intelligent" analysis for summer exam results.

Hope that helps.

Thank you Resin1 - most useful. I had been looking at SISRA online prior to my post - but I don't know much about 4matrix. I'll check it out.

Based on my research thus far, it seems to me that schools are looking for and would value a graphical/visual approach to data analysis - since most people receive 80% of thier information visually. Would you agree?

I do appriciate your time - but if you have another spare five minutes - please could you expound on the following statement:

"the assessment module should allow users to define means of recording grades against elements that have their (DCSF) performance measure contributions built in, so that the analysis part of the system can do all the hard work for the user, and allow the level of top level decision making for intervention etc that the above examples are currently providing."

Thanks in advance.

Quote:

---

Originally Posted by Kozan

What kind of assessment analysis do you presently do in your school?

---

Basic percentages-of-grades.

Quote:

---

What kind of trend/predicted analysis do you do?

---

I forget the exact name and algorithm at the moment, but something provided by one of the members of staff (who is a good statistician and knows what she's doing).

Quote:

---

How do you make use of “Value added” data such as FFT, Yellis, MidYis, Raiseonline?

---

I'm not sure.

Quote:

---

How much of this analysis is done in your MIS?

---

None.

Quote:

---

How much of the analysis done outside of your MIS – say in Excel?

---

The analysis is currently done by a set of Python scripts that takes in a set of raw figures and produces a bunch of tables and charts. This is flexible but not very maintainable (i.e. I know how it works, no-one else does). Unfortunatly, none of our teaching staff have enough knoweldge of Excel to be able to use it.

Quote:

---

How to you choose to present your data: Tables? Bar graphs? Line graphs? Colour coded conditional formatting?

---

Tables, assorted charts.

Quote:

---

How often is this analysis done?

---

Once a year, which much wailing and gnashing of teeth, although given more data updates the script could easily be run every night or however often is required.

Quote:

---

Who does this analysis in your school?

---

Me.

Quote:

---

What problems/difficulties do you encounter with your current analysis process?

---

Teaching and admin staff don't know enough about statistics to be able to properly articulate what it is they want to show, and if they did then they lack the IT skills to convert that articulation in to tables or charts. It should be perfectly possible to knock something together with Excel but they simply do not have the skills to do that.

Quote:

---

What features and functions would you like to see in assessment analysis module?

---

It needs to produce "nice looking" charts that we can use for marketing purposes. It needs to be simple to use, with no fuss, for people with few basic IT skills.

--
David Hicks

ok Kozan - quickly ;-)

yes - visual, visual visual

what I mean by that statement, is that I'd want to be able to set-up a progress tracking system where for example, I say for this subject we need to collect predicted grades that are full GCSE (A*, A, B, C etc) and I am able to chose an element that allows those types of grades to be input and within the system the element is classed as "1 whole GCSE" or in other words 20% contribution to Level 2 performance measure threshold (5+ A*-C) and that a A* is worth 58 QCA points. Similarly, I could assign a GCSE grade collection element for a GCSE short course, and the system would know it is 1/2 a GCSE/10% contribution/A*=29 etc. Or for an OCR National Level 2 Certificate, I could collect Pass/Merit/Distinction and it would know these are 4 GCSE/80% contribution/Distinction=220 points etc.

see Home Page - National Database of Accredited Qualifications if you are not familiar with all that stuff (search for a qualification and look at its performance measures).

Although I haven't used either of the those systems I mentioned, I believe what they provide is a mechanism to map subject name/raw grade data collected via MIS or other, to an internal qualification database that then means they can analyse that raw data, present it meaningfully and allow drill down by users for decision making, in a context within which those users are being judged by external parties.

Although I'm sure there will be a place in the market for them for some time to come and I'm not wishing them out of business (apologies if any of their reps look in here) but it would seem to me that an MIS should be able to offer something for assessment/progress monitoring that is designed and fit for the purpose(s) of a large number of users out there, rather than delivering something that is generic and may allow all to do something, but doesnt really fit a specific purpose.

I'm sure all these companies will appreciate you all helping someone copy their ideas!

hi kozan
worth making the module talk easily with external systems, and have flexible input and output of data. e.g. raiseonline, and fft data, both heavily used by ofsted.
possibly worth contacting those bodies as well.
there's also another body to do with ks2/3 grades that i made contact with some time ago, i forget the name, i think it's the people that send the CAT scores out, worth talking to them, the developer was most helpful to me, once i'd socially engineered his private mobile that is! :)
good luck.

Quote:

---

Originally Posted by PhilNeal

I'm sure all these companies will appreciate you all helping someone copy their ideas!

---

Can't harm to see what's on the market to see what you can Discover :troll::troll:

Quote:

---

Originally Posted by PhilNeal

I'm sure all these companies will appreciate you all helping someone copy their ideas!

---

Someone needs to pull of the ideas from these disparate systems and put them all in one system that schools can use effectivley, we cannot afford five systems that each do a little bit well. My MIS's analysis suite currently does not do half (or even a tenth if I am honest) the things I would wish, but I would be happy to swap if someone were to write one that did all the great stuff I have to do by hand in excel currently.

Students matter, products do not.

Quote:

---

Originally Posted by PhilNeal

I'm sure all these companies will appreciate you all helping someone copy their ideas!

---

That occurred to me too, Phil. Certainly, the one area where most MIS software is lacking is in providing ready-made analyses of the kind that schools actually want; marketing materials from the various vendors (Bromcom and Capita included!), alongside blanket questionnaires like this, suggest that this is unlikely to change in the immediate future.

I don't really want to write a full analysis suite spec for free, but I will give some tips:

• Have a read through the literature - there are many guides online to analysing assessment data available, which will give you an idea of typical foci of interest. The national strategies sites, forums such as this one, LA support sites, etc, have guidance on how to measure performance, and what to look for.
• Talk to some schools, i.e. go and visit them and see what they do - then you can think about how to solve the problems they're having to solve themselves at the moment
• Look through the feature requests for products like TeachersWebFolder - would any of these fit your brief?
• Have a read through DCSF and Ofsted guidance on how to assess the effectiveness of schools and LAs - this will give you an idea of the kind of questions schools will need to answer

It's going to be difficult to make anything easy for schools for as long as MIS software allows users to store assessment data the way they want. A lot of the time, schools tend to be very focussed on colour-coded lists of students: a quick search online will show you what they're asking about!

If this is a sanctioned project at Bromcom, you may want to look at getting in a few consultants (some who are "data-focussed" and some who are "anti-data") to discuss what they think would be helpful. Is this a Bromcom thing? I would imagine the firm is big enough to pay a team to develop a solution, and not just to let a new analyst "have a go" based on what they can find in a discussion forum...

Overall, the kind of questions you need to answer are:

• What's going wrong, and for which subject/students/groups of students?
• What's working, and (again) where is it working?
• How did we do in the past? How will we do in the future?
• Where do we need to improve in order to hit XYZ threshold target?

The above comments made me smile - especially Vik's - nice one.

Certainly dont see the need to defend my self, but I'll just say that any vendor worth their salt should be constantly reviewing and researching to see what other vendors or complimentary solutions are up to, in order to either stay one step ahead of the competition or try to at least deliver something that is competitive. Anything stated earlier could be gleaned within a few minutes perusal of their web-sites, which I'm sure, Kozan's marketing or product management colleagues would hit sooner or later if he didn't himself.

To me they were just examples of what users might demand. Heaven forbid that one vendor might be influenced by another commercial offering :rolleyes:

Hi

I am currently looking for some software that can produce this kind of analysis. I currently manually create analysis from Assessment Manager7 and spreadsheets. We are a large school and this can be an extremely time consuming process, especially now that we have to produce minority analysis along side the normal analysis.

We need software that can tell us:

Percentage A*-C in KS4 & 5 in each subject for Assessments, Targets & FFT or Alis
Percenatge 5+ & 6+ in Ks3 Subjects for Assessments, Targets & FFT

This data also needs further analysis to tell us how many of these are Girls,Boys,SEN,FSM,EAL & Ethnic Minorty Students.

Currently the process i'm using takes me two weeks to collate.

I'm sure i'm probably doing it in a much longer way than needed.

Can anyone tell me what software (if any) they use? Or how they use SIMs to generate such analysis that may cut my time spent doing our termly analysis.

Any help is appreciated.

Kelly

@Mathews: Our brilliant data manager has built a phenomenal spreadsheet that does this and more - it's mind-boggling to behold, but she can more or less export the data from sims and copy it in and boom - it all works. I can ask her if she will let me post it for you if you like, but be warned - it's MAAASSSSIVVE (and pink, she likes all her spreadsheets to be pink!!). :D

lol... Tyiell

I have tried to do a spreadsheet to house analysis and i can get one to do KS3 as our students in KS3 all do the same subjects. It gets more complicated in KS4 due to their option choices etc...

But yes please if you could post it or send me a copy i'd be enternally grateful!!!

I would ofcourse want to pay her (in biccies) for her trouble ;)

@Mathews - can't you use analysis and trend tracking tools in AM7?