The Analysis of Assessment Data in your school

A very interesting thread.

The responses to the initial question have shown that there is a demand for tools that can handle performance data in a more flexible way than is possible at present.
There are plenty of good ideas suggested here to be able to pull together a spec of the sorts of additional data tools that people would find useful.

I imagine that the astute performance data manager will need a range of tools - SIMS, RAISEonline etc. plus the enduring flexibility of Excel. One of the needs seems to be to reduce the time spent trying to move data from place to place, or on writing macros. That time could be spent working to help teaching staff to raise school standards.

I suspect that a small army of people in schools are busy trying to program Excel to do exactly the same tasks. It may be that EduGeek could provide an exchange service for data utilities that could save others time? Maybe it should encourage an Open Source approach to creating what is needed?

The good news is that MIS companies, and others, are creating additional applications that should make professional life more productive for those who handle data for a living.

Mike Bostock

Quote:

---

Originally Posted by MikeBostock

I suspect that a small army of people in schools are busy trying to program Excel to do exactly the same tasks. It may be that EduGeek could provide an exchange service for data utilities that could save others time? Maybe it should encourage an Open Source approach to creating what is needed?

---

I don't think a multi-user Excel-based software development project is really going to work out that well. Binary format spreadsheets don't lend themselves to working properly with source code control systems. Good, old-fashioned terminal applications are the easiest things to program - read some data in, churn it around a bit, write it out. Possibly a collection of command-line applications ready to be stuck together with Unix-style pipes?

--
David Hicks

The problem is a solution at the 'ideal' price - in our case, the immortal £free...
B

@Kozan (#1)

Great news! All competition is healthy etc.

Residuals are of prime interest imho.

Some analysis is done in SIMS etc but there are lots of homegrown EXcel apps out there, which represents a lot of duplicated effort for what is conceptually very simple.

Focus on what problem(s) you are trying to solve and that should guide you. Best way to get this is
to talk to as many potential end users as possible. I'm sure you can find a few in your customer base.

@resin1 (#2)

Agreed that MIS providers should be doing this. Is it even an MIS system if it doesn't?

@vikpaw (#8)

Agreed. Looking at what's out there at present is always the best place to start.

@skunk (#9)

"we cannot afford five systems that each do a little bit well" - especially if budgets get squished over the next few years, as seems likely.

@MattMitchell (#10)

If it does get outsourced, the team that developed a lot of reports and dashboards using SIMS 'business objects' layer for Rochester/CACI is still out there.

I can put Bromcom in touch.

@Tyiell (#13)

Do share!

@garrysaddington (#17)

"What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it".

Well said. Makes a mockery of the 'MIS' label really. The fact that the SIMS SQL Server schema (which is YOUR database) is 'undocumented' and supposedly off-limits for even read access is a major part of the problem. This is what needs to change, and quickly. That would have a dramatic impact on assessment analysis apps.

@MikeBostock (#31)

An open source style collaboration would save a lot of time that's currently spent duplicating each other's efforts. SourceForge.net provides a good framework for this approach.

@Setra (#33)

A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer!

I would be interested in any development that allowed us to compare assessment data with other schools in our cluster. Happy to use assessment manager and PA within our school but the cluster has a variety of MIS suppliers and assessment methods. Anyone come across a solution used more widely than single schools ?

Quote:

---

Originally Posted by NickBryson

I would be interested in any development that allowed us to compare assessment data with other schools in our cluster. Happy to use assessment manager and PA within our school but the cluster has a variety of MIS suppliers and assessment methods. Anyone come across a solution used more widely than single schools ?

---

Using assessment data across more than one institution is the way that Performance Management Systems need to go. Diplomas require movement of students between schools, and local authorities look at schools in relation to their contribution to area targets. At the macro level the development of SIF was designed to make data integration from more than one MIS type possible - but SIF is quite a slow-moving development with few practical solutions to this work yet in sight.

It is difficult for schools to use home-grown solutions for something as complex as this, but if data managers across several schools got together they could probably agree a common data format and provide extracted data using whatever methods are available to them. School level expertise in spreadsheets can go a long way towards providing useful comparisons. The approach would be to treat several institutions like one big school and add an additional tag to students so that students and courses from different schools can be compared.

There are many useful investigations that can take place on data arranged in this way. A school could compare the relative performance of diploma and traditional qualifications, the relative strengths of the same subjects taught in different schools, whether particular group of pupils do better or worse in different learning environments, etc.

It is certainly worth doing.

Mike what is SIF ?

Quote:

---

Originally Posted by DBMan

@Kozan (#1)

Great news! All competition is healthy etc.

+ + + + + +

@MattMitchell (#10)

If it does get outsourced, the team that developed a lot of reports and dashboards using SIMS 'business objects' layer for Rochester/CACI is still out there.

I can put Bromcom in touch.

@Setra (#33)

A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer!

---

To DBMan and all,


" A collaborative effort (see above) or an 'affordable' offering from a company like Bromcom may bring that reality closer! "

Thank you all your constructive input. The thread has been Kozan's own personal initiave. Generally Bromcom or any commercial company does not seek out input in this unorthodox means but nevertheless it appears to have developed into an interesting dialogue.

Yes, indeed -Bromcom endeavours to provide "affordable" offerings. We could be also open to proposals/ideas based on 'collaborative' effort principles.

Let's hear more on this and how you see it working in practical terms.

Quote:

---

Originally Posted by NickBryson

Mike what is SIF ?

---

Hello Nick.
I am not really an expert on SIF, but I know it to be the Schools Interoperability Framework. It is a system imported from the USA, where they have spent some years trying to make it possible to make use of information from many different databases.

Interoperability is the key to making better use of data in Education, for example, to inform professional staff in school about the relevant background of the pupils that they teach (on a need-to-know basis). It could bring together education records with social services records, police records - school and local authority information. I put together a presentation on this subject when I was involved with Naace, which sets out the landscape on interoperability issues and can be seen at http://www.new-media-learning.com/interoperability

SIF is a sophisticated (read 'complex') method for conveying data in a standardised encrypted format. Becta has been leading on its developing in the UK, but it seems to me that it will be a very long time before it provides any practical solutions of the type that have been discussed on this thread. The pragmatic approach that we have taken in our work is to simply open as many data channels as possible to importing and exporting information. So, for example we make sure data can be imported as XML, XLS, CVS, JPG etc and exported as XML, XLS, DOC, PDF etc. and as far as possible implement copy and paste to do this. If SIF ever appears we can add that too.

Developments in using data in Education are so far behind the way that data is used in other professional walks of life that we can't wait for SIF to bring solutions to this problem, especially if it can improve life for education and data professionals - and the life chances of pupils in schools.

There seems to be a nice step away from the static "office" style UI in products, towards a more freeflowing, visual, interactive way of manipulating, extracting, and displaying vast amounts of data quickly. At a glance get an overall picture/trends, then drill down to the detail if/when required.

Microsoft have a research project called Pivot:

Learn More about Pivot | Live Labs Pivot

Quote:

---

What is the point of collecting all the data in a MIS to then have to export it somewhere else to do something useful with it. Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS.

---

Because some people:
can't remember the password
have never come across a tabbed workbook
don't know how to scroll accross a spreadsheet and so think the column they need is missing
only remember to include every member of their group if the whole class is listed together
fill down on a filtered sheet
sort only the bit of the sheet they care about
think that the data validation lists are wrong and have enough expertise to overwrite it and put in the 'right' type of data
save corrupted sheets with the original title
or versions of it with thoroughly uninformative names (having been confused by request for a second password) ...

OK, with appropriate protection, training & extra columns none of those are a big issue, but we collect and analyse data in Excel every term (and then some, thank you National Challenge) and all of the above are issues that crop up time & time again ... and that's with one book per faculty (I've not got time to generate one per teacher). I dread to think what would happen if I let 60+ staff loose in a workbook big enough to hold everything.

So I'm hoping the opposite: that we'll soon be able to collect the data through SIMs because it is much better than I am at asking individuals for exactly the information they need to supply and preventing people muddling things up. OK it will take a lot of time to set up, but so far I've found exporting info from SIMs relatively painless and so will still be using Excel for most of the analysis because of the issues listed by JoshJohnson.

PS Which is a shame, because some of the tools in Performance Analysis are quite nifty.

Quote:

---

Originally Posted by CaterinaAnna

Because some people:
.

---

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

Quote:

---

Originally Posted by garrysaddington

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

---

That statement sounds a bit like Charles Duell's quote in 1899 that "Everything that can be invented, has been invented." MIS suppliers and the smaller companies that produce data-related applications are providing choice in this marketplace. It is unlikely that one company can provide everything that the market demands, so it is important that data can be made to move easily between applications.

Imagine if Microsoft tried to exercise a monopoly and force people to only ever use text in Microsoft Word. There is so much that can be done with text that it is obvious that we must be able to cut and paste it into as many applications as covers the range of functions that we need. We are starting to think more creatively about performance data and this will stimulate a healthy desire for further data tools.

Quote:

---

Originally Posted by MikeBostock

That statement sounds a bit like Charles Duell's quote in 1899 that "Everything that can be invented, has been invented.".

---

I fail to see the relevance of this statement to what I am trying to say.
I think you misunderstand too. There are a series of guidelines laid down by government bodies that give recommendations on what a MIS should do, not what is there already. If the MIS market was sound then we would not need all these add on products that add extra complexity and financial burdens on schools. In my opinion a healthy desire for further data tools means that tax payer's money is being wasted in the first instance on tools that are not doing the job they purport to do. There should be no need for choice in this market.

Quote:

---

Originally Posted by garrysaddington

I think you may have missed my point. If the MIS conformed to DCSF/BECTA guidelines, all this talk of export to Excel would be superfluous because the MIS would do all the data analysis for you.

---

Probably should have snipped more: I was really responding to the part 'Why not have a spreadsheet open in the staffroom to collect the data and cut out the MIS? Then ask for a reduction in annual fees for your MIS. ' which I appreciate may not have been entirely serious.

I agree entirely that the MIS should be able to collect the data, carry out the analyses we need and any others we could possibly think of - and it's that latter part that AM7 still falls down on ... and I suspect always will do. Whatever analysis and reporting tools are built into the MIS, someone somewhere sometime will demand an analysis or way of presenting data that the MIS developers haven't thought of and could only be got if the system included all the tools of the latest office suite.

I'd rather have a reliable, well-structured database that will produce the most commonly needed analyses easily, has a user-friendly front end and proper access to all the data than something that tries to be all things to all men & ends up frustrating people because it can't do things they're used to being able to do in excel or word.