The Analysis of Assessment Data in your school

[willkeddie]

Tyiell - I am currently investigating ways of doing this. I would really appreciate a copy of that spreadsheet if at all possible. I could send some biccies also - but they may go a bit soft in the post!

[Dancer]

As a SIMS user I feel that SIMS actually does quite lot of analysis within both Assessment and Performance.

Aspect Analysis quickly calculates A*-C etc for every subject in any assessment

Resultset anlysis allows the tracking progess within a subject across many assessments

Group analysis allows access to many different groups

Chance analysis is great for monitoring levels progress, stalling students, comparing against targets etc.

 

My main complaint that it is essentially me who prints out a load of tables from performance and distrubute them to HOD/SLT etc after each assessment and that it would be very difficult to extend its usage to the average person who uses the data. This undermines the data mining capabilities. I'm starting to look at tracking grids which means that much more is setup before hand but are relatively easy for the relevent HOD/SLT to run.

 

And the wierd filters where some fields are available in some filters and not in others (I think already been mentioned in this thread). You can set up a group analysis and then filter the groups on SEN Status but not set a SEN Status as a group. If I remember correctly you have different fields if you set the group filter on a marksheet template and in an indivdual template.

[ChrisH]

I am like DHicks but my routines are in VBA and only I know how to use them as they are a little clunkey. I have programmed custom functions with usually take columns on another worksheet and one or two filters like FSM and Male or SEN and not gifted etc. I then populate the various tables with these functions. I have some nice short routines that will work out who has 5 or more A* to Cs and if that includes maths & english etc. The core is pretty flexible so if anyone has any particular requirement for a macro for excel then let me know and I will see what I can do.

[CAM]

We use a modified version of the KS5 ALPS system. It is exported from Assessment Manager to Excel, residuals are generated and then I use a custom-built VBA script to apply a 4 colour traffic light color scheme to the grades. At the side is the total residual per student and it's sorted from high to low. At the bottom is the %A*-C percentages and the difference in the total points values between the current set of collected grades and the targets for each subject.

[Setra]

1. What kind of assessment analysis do you presently do in your school?

 

For example do you produce: Student/Subject Residuals, % of students achieving certain grade ranges for the core subjects?

Chances, On Target, Residuals, Range achievements A*-A, 5+ A*-C with Ma & En, 5+ A* - C, Achieving 5+ A* - G, Achieving 1+ A* - C, Achieving 1+ A* - G

All with Gender splits, Class splits, SEN, FSM, Ethnic, Language and Looked After as well as whole subject and whole year all requiring different aspects.

 

2. What kind of trend/predicted analysis do you do?

Very little at present; conditionally formatted spreadsheets, mostly, little grapic analysis.

 

3. How do you make use of “Value added” data such as FFT, Yellis, MidYis, Raiseonline?

We only use FFT and we use it equally with Teacher Assigned targets.

 

4. How much of this analysis is done in your MIS?

The A-Level chances are. Mostly.

But they are then exported to Acrobat to be coloured in - they're obsessed with traffic lights here...

 

5. How much of the analysis done outside of your MIS – say in Excel?

90%

 

6. How to you choose to present your data: Tables? Bar graphs? Line graphs? Colour coded conditional formatting?

Tables with traffic light formatting.

Chance matrices with traffic light formatting.

 

7. How often is this analysis done?

It is done for every year for every report - so 7 times a year.

The there's the big Exam Results stuff in the Summer. I've not been there yet...

 

8. Who does this analysis in your school?

Me! Except 3 of the Maths teachers help do some of the summer stuff (adding in the previously attained AS and GCSE grades) because of the short timeframe.

 

9. What problems/difficulties do you encounter with your current analysis process?

Massively time-consuming, esp. if splitting by calss - we have maybe 120 classes in each year.

Summer Exams only accept that season's results, so GCSE and As levels taken in previous years are not included in SIMs-generated analysis - hence the Maths teachers.

I'm currently working on a MAHOOSIVE spreadsheet to accomodate those. Apparently.

 

10. What features and functions would you like to see in assessment analysis module?

Choose the results to include. Colour-coding. Quickly flick between class and whole year analysis.

Also, pre-set templates to prevent creating a billion - one for each subject each slightly different.

 

Hope it helps.

B

Edited April 30, 2010 by Setra

[Guest MikeBostock]

A very interesting thread.

 

The responses to the initial question have shown that there is a demand for tools that can handle performance data in a more flexible way than is possible at present.

There are plenty of good ideas suggested here to be able to pull together a spec of the sorts of additional data tools that people would find useful.

 

I imagine that the astute performance data manager will need a range of tools - SIMS, RAISEonline etc. plus the enduring flexibility of Excel. One of the needs seems to be to reduce the time spent trying to move data from place to place, or on writing macros. That time could be spent working to help teaching staff to raise school standards.

 

I suspect that a small army of people in schools are busy trying to program Excel to do exactly the same tasks. It may be that EduGeek could provide an exchange service for data utilities that could save others time? Maybe it should encourage an Open Source approach to creating what is needed?

 

The good news is that MIS companies, and others, are creating additional applications that should make professional life more productive for those who handle data for a living.

 

Mike Bostock

Edited May 8, 2010 by vikpaw

[dhicks]

I suspect that a small army of people in schools are busy trying to program Excel to do exactly the same tasks. It may be that EduGeek could provide an exchange service for data utilities that could save others time? Maybe it should encourage an Open Source approach to creating what is needed?

 

I don't think a multi-user Excel-based software development project is really going to work out that well. Binary format spreadsheets don't lend themselves to working properly with source code control systems. Good, old-fashioned terminal applications are the easiest things to program - read some data in, churn it around a bit, write it out. Possibly a collection of command-line applications ready to be stuck together with Unix-style pipes?

 

--

David Hicks

[Setra]

The problem is a solution at the 'ideal' price - in our case, the immortal £free...

B

[DBMan]

@Kozan (#1)

 

Great news! All competition is healthy etc.

 

Residuals are of prime interest imho.

 

Some analysis is done in SIMS etc but there are lots of homegrown EXcel apps out there, which represents a lot of duplicated effort for what is conceptually very simple.

 

Focus on what problem(s) you are trying to solve and that should guide you. Best way to get this is

to talk to as many potential end users as possible. I'm sure you can find a few in your customer base.

 

@resin1 (#2)

 

Agreed that MIS providers should be doing this. Is it even an MIS system if it doesn't?

 

@vikpaw (#8)

 

Agreed. Looking at what's out there at present is always the best place to start.

 

@skunk (#9)

 

"we cannot afford five systems that each do a little bit well" - especially if budgets get squished over the next few years, as seems likely.

 

@MattMitchell (#10)

 

If it does get outsourced, the team that developed a lot of reports and dashboards using SIMS 'business objects' layer for Rochester/CACI is still out there.

 

I can put Bromcom in touch.

 

@Tyiell (#13)

 

Do share!

 

@garrysaddington (#17)

 

"What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it".

 

Well said. Makes a mockery of the 'MIS' label really. The fact that the SIMS SQL Server schema (which is YOUR database) is 'undocumented' and supposedly off-limits for even read access is a major part of the problem. This is what needs to change, and quickly. That would have a dramatic impact on assessment analysis apps.

 

@MikeBostock (#31)

 

An open source style collaboration would save a lot of time that's currently spent duplicating each other's efforts. SourceForge.net provides a good framework for this approach.

 

@Setra (#33)

 

A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer!

[NickBryson]

I would be interested in any development that allowed us to compare assessment data with other schools in our cluster. Happy to use assessment manager and PA within our school but the cluster has a variety of MIS suppliers and assessment methods. Anyone come across a solution used more widely than single schools ?

[Guest MikeBostock]

I would be interested in any development that allowed us to compare assessment data with other schools in our cluster. Happy to use assessment manager and PA within our school but the cluster has a variety of MIS suppliers and assessment methods. Anyone come across a solution used more widely than single schools ?

 

Using assessment data across more than one institution is the way that Performance Management Systems need to go. Diplomas require movement of students between schools, and local authorities look at schools in relation to their contribution to area targets. At the macro level the development of SIF was designed to make data integration from more than one MIS type possible - but SIF is quite a slow-moving development with few practical solutions to this work yet in sight.

 

It is difficult for schools to use home-grown solutions for something as complex as this, but if data managers across several schools got together they could probably agree a common data format and provide extracted data using whatever methods are available to them. School level expertise in spreadsheets can go a long way towards providing useful comparisons. The approach would be to treat several institutions like one big school and add an additional tag to students so that students and courses from different schools can be compared.

 

There are many useful investigations that can take place on data arranged in this way. A school could compare the relative performance of diploma and traditional qualifications, the relative strengths of the same subjects taught in different schools, whether particular group of pupils do better or worse in different learning environments, etc.

 

It is certainly worth doing.

[NickBryson]

Mike what is SIF ?

[BromcomPublicRelations]

@Kozan (#1)

 

Great news! All competition is healthy etc.

 

+ + + + + +

 

@MattMitchell (#10)

 

If it does get outsourced, the team that developed a lot of reports and dashboards using SIMS 'business objects' layer for Rochester/CACI is still out there.

 

I can put Bromcom in touch.

 

@Setra (#33)

 

A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer!

 

To DBMan and all,

 

 

" A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer! "

 

Thank you all your constructive input. The thread has been Kozan's own personal initiave. Generally Bromcom or any commercial company does not seek out input in this unorthodox means but nevertheless it appears to have developed into an interesting dialogue.

 

Yes, indeed -Bromcom endeavours to provide "affordable" offerings. We could be also open to proposals/ideas based on 'collaborative' effort principles.

 

Let's hear more on this and how you see it working in practical terms.

Edited May 8, 2010 by vikpaw

[Guest MikeBostock]

Mike what is SIF ?

 

Hello Nick.

I am not really an expert on SIF, but I know it to be the Schools Interoperability Framework. It is a system imported from the USA, where they have spent some years trying to make it possible to make use of information from many different databases.

 

Interoperability is the key to making better use of data in Education, for example, to inform professional staff in school about the relevant background of the pupils that they teach (on a need-to-know basis). It could bring together education records with social services records, police records - school and local authority information. I put together a presentation on this subject when I was involved with Naace, which sets out the landscape on interoperability issues and can be seen at http://www.new-media-learning.com/interoperability

 

SIF is a sophisticated (read 'complex') method for conveying data in a standardised encrypted format. Becta has been leading on its developing in the UK, but it seems to me that it will be a very long time before it provides any practical solutions of the type that have been discussed on this thread. The pragmatic approach that we have taken in our work is to simply open as many data channels as possible to importing and exporting information. So, for example we make sure data can be imported as XML, XLS, CVS, JPG etc and exported as XML, XLS, DOC, PDF etc. and as far as possible implement copy and paste to do this. If SIF ever appears we can add that too.

 

Developments in using data in Education are so far behind the way that data is used in other professional walks of life that we can't wait for SIF to bring solutions to this problem, especially if it can improve life for education and data professionals - and the life chances of pupils in schools.

Edited May 8, 2010 by MikeBostock

[teak]

There seems to be a nice step away from the static "office" style UI in products, towards a more freeflowing, visual, interactive way of manipulating, extracting, and displaying vast amounts of data quickly. At a glance get an overall picture/trends, then drill down to the detail if/when required.

 

Microsoft have a research project called Pivot:

 

Learn More about Pivot | Live Labs Pivot

[CaterinaAnna]

What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it. Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS.

 

Because some people:

can't remember the password

have never come across a tabbed workbook

don't know how to scroll accross a spreadsheet and so think the column they need is missing

only remember to include every member of their group if the whole class is listed together

fill down on a filtered sheet

sort only the bit of the sheet they care about

think that the data validation lists are wrong and have enough expertise to overwrite it and put in the 'right' type of data

save corrupted sheets with the original title

or versions of it with thoroughly uninformative names (having been confused by request for a second password) ...

 

OK, with appropriate protection, training & extra columns none of those are a big issue, but we collect and analyse data in Excel every term (and then some, thank you National Challenge) and all of the above are issues that crop up time & time again ... and that's with one book per faculty (I've not got time to generate one per teacher). I dread to think what would happen if I let 60+ staff loose in a workbook big enough to hold everything.

 

So I'm hoping the opposite: that we'll soon be able to collect the data through SIMs because it is much better than I am at asking individuals for exactly the information they need to supply and preventing people muddling things up. OK it will take a lot of time to set up, but so far I've found exporting info from SIMs relatively painless and so will still be using Excel for most of the analysis because of the issues listed by JoshJohnson.

 

PS Which is a shame, because some of the tools in Performance Analysis are quite nifty.

[garrysaddington]

Because some people:

.

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

[Guest MikeBostock]

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

 

That statement sounds a bit like Charles Duell's quote in 1899 that "Everything that can be invented, has been invented." MIS suppliers and the smaller companies that produce data-related applications are providing choice in this marketplace. It is unlikely that one company can provide everything that the market demands, so it is important that data can be made to move easily between applications.

 

Imagine if Microsoft tried to exercise a monopoly and force people to only ever use text in Microsoft Word. There is so much that can be done with text that it is obvious that we must be able to cut and paste it into as many applications as covers the range of functions that we need. We are starting to think more creatively about performance data and this will stimulate a healthy desire for further data tools.

[garrysaddington]

That statement sounds a bit like Charles Duell's quote in 1899 that "Everything that can be invented, has been invented.".

I fail to see the relevance of this statement to what I am trying to say.

I think you misunderstand too. There are a series of guidelines laid down by government bodies that give recommendations on what a MIS should do, not what is there already. If the MIS market was sound then we would not need all these add on products that add extra complexity and financial burdens on schools. In my opinion a healthy desire for further data tools means that tax payer's money is being wasted in the first instance on tools that are not doing the job they purport to do. There should be no need for choice in this market.

[CaterinaAnna]

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

 

Probably should have snipped more: I was really responding to the part 'Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS. ' which I appreciate may not have been entirely serious.

 

I agree entirely that the MIS should be able to collect the data, carry out the analyses we need and any others we could possibly think of - and it's that latter part that AM7 still falls down on ... and I suspect always will do. Whatever analysis and reporting tools are built into the MIS, someone somewhere sometime will demand an analysis or way of presenting data that the MIS developers haven't thought of and could only be got if the system included all the tools of the latest office suite.

 

I'd rather have a reliable, well-structured database that will produce the most commonly needed analyses easily, has a user-friendly front end and proper access to all the data than something that tries to be all things to all men & ends up frustrating people because it can't do things they're used to being able to do in excel or word.

Edited May 12, 2010 by CaterinaAnna
additions having read posts between quoted & reply

[vikpaw]

Does anyone actually make good use of the Performance Analysis module in SIMS? i mean really good use. It's always been on the back burner with me, as there is always something more urgent or critical that needs fixing / patching etc.

 

As for replicating things you can do in Excel and other apps, that would be great. it always feels like we are being kept out of reach of our data by the user interface and the way we must access the numbers.

[PhilNeal]

VikPaw, yes they do and its incredibly powerful!

 

I don't think it is as easy to use as it could be (hence Discover) however on the Capita-CS.UK.CO site take a look at the Norbury School Talking Head (search for "Norbury"). This is a middle school head talking about her use of it. Equally take a look at the "Featherstone" case study, a secondary school that has achieved "outsanding" status and attributes much of their success to PA.

[PhilNeal]

Do you mean like your MIS supplier should have already been doing, without making schools do the work. See the link below (from 2008).

 

Management Information Systems (MIS) and pupil tracking: Developers' guidance

 

What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it. Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS.

 

The document that you refer to was written in co-operation with the MIS suppliers i.e. we practically wrote it! SIMS AM & PX does all of the things that are listed in the guide without exporting to a spreadsheet.

[Sivadam]

The document that you refer to was written in co-operation with the MIS suppliers i.e. we practically wrote it! SIMS AM & PX does all of the things that are listed in the guide without exporting to a spreadsheet.

 

What a super comment Phil!

[garrysaddington]

The document that you refer to was written in co-operation with the MIS suppliers i.e. we practically wrote it! SIMS AM & PX does all of the things that are listed in the guide without exporting to a spreadsheet.

 

If that is the case why is there so much discussion on here about using spreadsheets. I only base my comments on the content of these forums.