Extracting data from an MIS

[GeorgeFaux]

Phil Neal... I agree that SIMS and CMIS has the tools to do a more diagnostic analysis, the problem is the mindset of staff who either do not know how to pull the data out or how to use it properly, the ability to change things in mid flow is difficult and goes against instinct. Fisher, Raise, and MIS data are usually used to sit in a meeting and say this is the situation, no further.

 

Limbo we did the same thing using vb scripts to migrate data into excel sheets, we did the sessional attendance and it worked beautifully. Yes SQL is easier to work with but I do think that if you are having to go third party you may as well make your own MIS. Its a real failing we had to do that.

[GrumbleDook]

@George and Mike

 

Both of you fulfill the role of an Analyst for schools. You get them the information they need and help them understand it. That raises theconcern I mentioned about about schools relying on others to understand things they should understand themselves.

 

However ... we should remember that using consultants and tools are necessary for a number of schools. It is what can help bring schools up to speed and make them 'good' in their use and understanding of data rather than just 'satisfactory' (or as we were ... we got the dreaded '4' for our use of data). We have worked on this and are now 'Good' but it has been a lengthy understanding process. Some schools cannot take this time or do not have the staff to do it.

 

Then you have the 'Excellent' schools ... schools that have got it right and have do it themselves ... and the schools that have got it right, but ahve made the choice to continue using tools as it fits in with them. (see similar threads about schools with RM, or other middleware, not just being about covering for no techies, but being used to free techies up to do other things)

 

Getting back to the original idea on this thread though ... are people happy about the prospective processes (or even present ones) for getting data out of MIS and if not what would be their preffered options (remember ... these are commercial companies so asking to go completely Open Source is not exactly going to be the most helpful answer ... )

[GeorgeFaux]

I think a lot of the misunderstandings come from the days when there was only really original SIMS on the market and the data was far less powerful. It is only in the last five or so years that using data heavily at an individual school level has really taken off. A lot of the staff who use the system are very usure when confronted with CMIS or .Net as the flexibility goes against what they know. When I have been into schools directly after a change over the first thing that people want is the new system to do exactly what the old one did before they see anything else.

 

However again I'd stress that it is not usually the MIS which fails to give the required data but more the interventions that arise from it. The student who is predicted a U in year 10 on a course is allowed to sit in a class for 2 years in a bottom set when it is obvious that they will fail at the end of year 11.

 

The mindset should be, can we get him a G, with extra tuition, with a different teacher, with after school help. If not could we get him a better grade in x if he was removed and allowed to swap bands or go to a different class. What is holding the child back, is it attendance, behaviour, punctuality, relationships. How can we solve this. Can we provide a Teaching assistant, a Learning Mentor, an academic coach. Should we involve SEN, EWO or other agencies. Is there an alternative vocational qualification.

 

The mere fact that there is a U anywhere should automatically flag up this conversation but it should also flag up. Why has the student got to year 10 in this condition, why did we not teach them in a different style earlier, who were their teachers, is there an issue there, who were they taught with, would mixed ability be better, what was the learning style used, have they had ongoing pastoral problems.

 

To every one of those questions there should be a possible answer, the only thing I am interested in getting out of a management system is who is failing/succeeding, why, and they once an intervention is in place what are we doing about it. The point was raised about outside consultants, I agree that the lack of this type of mindset keeps people like myself in business but I do see a change coming, I see a fair number of non teaching positions designed to do the above in the £20-35k.

 

I hope that MIS systems develop to include individual student intervention tracking, Flagging systems which identify failing students, personalised learning information, information and tests to determine learning styles and pastoral summaries.

[Kained]

Extracting the data in CMIS is easy manipulating it to usefulness is a pain.

[limbo]

As one of our SMT says "There is no point weighing a cow every six weeks if you are not actively doing something to help him loose / gain weight"

 

Abstract - but there is a point in there somewhere.

 

I got asked to develop a new system for one of our management teams to pull out data and use it to band students into different groups depending on whether they were on track of not based on our assessments. There was a lot of conversation about what triggered somebody being in which group and then, to try and help get some clarity I asked how we are going to then react to each group to see if this led to any obvious ways to group them. They have not got as far as that yet - so they are now thinking about different strategies to raise achievement and they will then decide on the profile of student that will benefit from it.

 

My Point? The system is usless without somebody thinking about the process and as such is often the last step - no the first.

[Guest MikeBostock]

Becta tell me that the Schools Interoperability Framework SIF is the only way to ensure that schools would get their pupil records reliably from their MIS if the wish to use Excel or third party tools to process their data.

 

At present the ability of schools to do their own research using contextual data stored in an MIS depends on having IT people with the skills to solve the problems of exporting data that most MIS have in one shape or form.

 

To make SIF export pupil records reliably from an MIS the MIS supplier has to create a SIF channel and supply an agent that decodes the standard exported SIF file into the pupil record.

 

It sounds a long way round and will take time to implement I suspect.

[gskelton]

Hi,

 

I'm new to eduction. I've worked in IT in business all my life so far. We have CMIS at our school. It seems very complex to me. I'd at least like to have a look at a data dictionary for the system so I'd be able to find out where data resides on the system.

 

Does anyone know where I could get my hands on such a nugget of informaion??

[mark80]

Hi,

 

I'm new to eduction. I've worked in IT in business all my life so far. We have CMIS at our school. It seems very complex to me. I'd at least like to have a look at a data dictionary for the system so I'd be able to find out where data resides on the system.

 

Does anyone know where I could get my hands on such a nugget of informaion??

 

Would love this!!

[StArbiter]

Hi,

 

I'm new to eduction. I've worked in IT in business all my life so far. We have CMIS at our school. It seems very complex to me. I'd at least like to have a look at a data dictionary for the system so I'd be able to find out where data resides on the system.

 

Does anyone know where I could get my hands on such a nugget of informaion??

 

Would love this!!

me too - yes please !

[johnymac]

We have finished the tool to extract the data from SIMS for primary education and it is due to go to public release soon.

 

It will take me a few hours max to do secondary.

 

I need to know how much demand there is for CMIS I guess, all of our clients at current SIMS users (but I imagine this will change).

 

If anyone has a CMIS server up there is a few things I need to make my life easier.

 

If someone can load profiler - set up a recording and run a query in CMIS on a year group I will be able to see if it uses a stored procedure or a plain old SQL query. That way I can plan creating the executable to harvest the data.

 

Also I'm assuming CMIS has an "Use Date" value, I need to know how this is formed.

 

If you have much demand for the SIMS client then drop me an email and I will spend some time with you getting it right for secondary education use. Any schools local to Bradford would be easiest but I can remote in/work over email/phone pretty easily.

 

If anyone on this forum wants to post a comprehensive list of requirements checked against this list then feel free. At current the client does this:

 

Queries MIS and exports:

pupil name - pupil class

teacher name

 

This data is the encrypted and either ftp'd up to a third party server or transfered locally

 

The data is imported every 24 hours into OU in this format:

School Name

- Teachers

- Pupils

- Admin/SMT

 

Security and Distribution groups are made for:

Teachers (sec.%firstthreelettersofschool%.teachers)

Each Class group (in format sec.%firstthreelettersofschool%.%classname%)

- Remvoe the sec. bit for distribution groups

 

Teachers are imported as firstname.surname

 

Pupils are imported as %firstthreelettersofschool%.firstname.surname

*** This is important for child protection so kids don't have [email protected] email address'

 

Exchange Values are also set in Exchange Environments for Query Base DN as some schools do not wish pupil a to be able to find pupil b in their GAL/OAB.

 

Accounts are enabled and set to never expire (optional)

 

Passwords are set to a user input - seperate for teachers and pupils, this is only done on user creation (1 time)

 

Pupils automaticaly change class if their class in MIS changes.

 

Teachers must be manualy placed into Admin/SMT OU.

 

 

 

 

---- End of specs.

 

 

 

 

I don't check the forum much so I recommend emailing me at [email protected] with any queries you may have.

[StArbiter]

Johny - this sounds good - how does it extract SIMS data and pick up the class / reg group changes ? How much does it cost for a primary ?

 

PS if you elect to watch the thread (or notify me) you will be emailed when there is a response !

[johnymac]

evidently

 

Its part of an email product we develop called primary email, we intend to make it free as an open source project as soon as its a final stable release - we have a few hundred schools using it at current with only a few minor issues that the clients haven't even noticed but I know about.

 

It extracts the data from SQL not SIMS, we avoid sims completely. As for class/reg group changes basically we create sec/dissy groups based on the value from a pupil so if a new class is added we automatically generate a sec/dissy group. When a user is moved group we just see the new group and remove the user from its old groups and stick it in the new groups.

 

Any more Q's?

[DoubleFirstChair]

Having seen the trauma experienced in users wanting to extract data from an MIS, I am pleased to say that we have written a Reporting System for hebron.NET which allows ANY and ALL data to be selected and sent to Excel.

 

Hopefully this will be found helpful.

[Muhaimin]

SIF will allow the flow of data to be made easier both vertically and horizontally

 

SCHOOL <> LA <> DCSF

 

SCHOOL <> SCHOOL (CTF already in place)

 

However, SIF will not make it easier to pull data out of MIS systems instead it will hopefully make the MIS system hold more current information that the LA or DCSF hold and have transferred to your MIS or vice versa.

 

Isn't the answer to contact your MIS supplier, put in a development request for the specific report you need, get given an issue number for the development and then in agreement with your supplier the report/export should be avilable through a single button (maybe additional options for criteria) within the agreed time frame - lets say aproximately 3.5 months or less.

 

Simple.

[olcinm]

SIF will allow the flow of data to be made easier both vertically and horizontally

 

SCHOOL <> LA <> DCSF

 

SCHOOL <> SCHOOL (CTF already in place)

 

Simple.

 

Except there needs to be agreement by all 5 or so MIS providers on the UK SIF Data Model. Last thing I heard was this wasn't going well. A sticking point could be that (if true) Capita claim only their MIS holds temporal data and so moving data by SIF would lose this temporal data connection.

 

Isn't this also true with CTF files anyway?

 

Could a UK SIF data model not transfer temporal data? Of course it would be useless going into a rivals DB unless they too have fields for this kind of temporal data. Of course SIMS<>SIMS would be no problem.

 

If the DCSF want temporal data then they'd have to use a DB that used such fields too. Can't see the problem myself.

 

Anyone know if any of this is true?

 

How important is the temporal data? if I was a teacher in KS2 taking in pupils from KS1 I'd probably not be interested in their various paths through particular subject NC levels but just their current level/sublevel . Or would I?

 

Capita appear to be a bit anti SIF i.e.

 

- I have been told by someone at Capita that SIF costs for each transaction made so it's the wrong choice economically.

- They would have preferred IMS was used as they'd already done development on IMS Enterprise and a few assessment items.

 

But they were keeping irons in the fire i.e.

 

- their involvement in the Birmingham PoC for SIF

- they may be looking to implement SIF for 14-19 MIS interoperability (e.g. Schools<>Sixth Form) for diploma work by creating a ZIS server and presumably working on SIF agents.

 

i have also heard that EduStructures the (only) company, so I'm told, in the UK with a ZIS on the market have got into bed heavily with Serco (CMIS) and may not be able to work with Capita.

 

And what are Microsoft doing with SIF and their BIZ Server?

 

Don't we need the DCSF/Becta and these guys to get this sorted!

 

Has anyone out there in school-land had a go at doing their own SIF ZIS to get data interoperability between databases? Is it that hard?

[PhilNeal]

Olcinm, you have summarised our position quite well but I will pick up on a few points.

 

CTF is designed to move data from one school to another at change of key stage. SIF is designed to be used day in day out so the temporal issue is important. We do keep dates against most items names, address, assessment etc etc. It is easy for a non temporal system to receive data from SIMS but if data is coming into SIMS do we assume that the relevant date is today or do we drop it into suspense and ask the operator to enter it?

 

A school’s data needs are different to the DCSF. The DCSF take snap shots of data for statistical purposes – it is not a working database. We need to show where a child’s address is now. Our pupil tracking system needs to assign a date to a reading test. In our view the temporal aspect is important and in any case that is how SIMS is designed and changing it is just not practical!

 

We are working with Becta and the other suppliers to refine the SIF data model for the UK but it is not ready yet. For example a child’s attendance data is not sent as marks against a day, it is sent as the number of present marks in a period. Our central system needs to know exactly when a child was present. We are helping to define the assessment data needs in a second proof of concept.

 

We do think that IMS is a much simpler and cheaper means of getting data from an MIS to a VLE and it is available in SIMS today at no extra cost. We are working with UniServity, Viglen, Granada, Fronter and RM to refine this solution. There are no per pupil charges and it is simple to set up.

 

We do not understand the full cost of SIF and if you look at the Becta report on SIF by PWC I think that comes out. What will the pupil/year cost be that a school has to pay to transport the data? How much will it cost to administer – the proposed model envisages the LA running the ZIS server for all of its schools. Suppose every school has 5 separate applications that are linked by SIF – would that require an extra member of staff in the LA to manage the 1250 or so links?

 

I could write a great deal more but I won’t!

[olcinm]

Hi Phil

 

Thanks for the detailed reply. Much food for thought there.

 

Regarding the point about CTF issues being the same as SIF issues wrt temporal data. If we had SIF we wouldn't need CTF would we? We'd have a SIF agent for school transfer I would have thought. A backup method of transfer perhaps for transfer to schools not in the SIF zone. A possibility of SIF would be the possibility of an LA MIS being unified with their schools so it could help with all sorts of business needs.

 

Regarding the DCSF snapshot requirement. Won't this change is they have an LA aggregating zone and ZIS? They could snapshot at their end. If the DCSF want roll-backable snapshots then don't all MIS systems have to provide this function?

 

Regarding the management of SIF how does the overhead of SIF (agent ?) link management differ from the management of IMS connectors?

 

Regarding your IMS data model. How complete is it? Why are other LP and MIS providers not working with you notably NetMedia, Serco and Pearson?

 

Do you use IMS for your SIMS Learning Gateway Webparts. If not why not? Wouldn't it be better to have a data interoperability solution that worked with Sharepoint and perhaps creating an intermediary subset of data on Sharepoint. Get the VLE's and your web parts talking to that. You could then allow for data connections going down between MOSS 2007 and SIMS .net

 

The Becta comparison of standards put SIF on top and IMS came second. IMS did not beat SIF in any criteria. Has Capita done a similar comparison of standards or is the IMS preference because you need to recoup development costs on the IMS route you've taken rather than a considered comparison? I'd like to see the report if you have. The Becta one is available here Becta Strategic Case for SIF

 

And why have I heard your proposed 14-19 Partnership Exchange server will be using SIF?

 

Fronter's IMS connectors are relying on a data dump to an intermediary server are they not? And this is mainly for provisioning of users and groups not assessment data.

 

Where do I find the document that states SIF costs per transaction? The evidence from the USA states that SIF leads to cost reductions.

 

A small innovative company that stores learner data will have to do a lot of development work with each MIS and LP and other service provider, knowing all their IMS implementations and maybe being charged for the privilege of writing data to the MIS. SIF cuts their costs down. Small innovative companies are vital to ensuring we get innovative development as is open source. I wonder if IMS will lead to bigger companies dominating even more.

 

Isn't one of the big issues with IMS is the lack of a definite transport mechanism leaving the doors open for incompatabilities re: SOAP and other web service standards and yet more work for the small innovator?

 

Another issue is that IMS is not specifically for schools so there's more work there too.

 

For Moodlers I believe IMS is already present for importing users/groups by file import but there isn't a web service for it yet but it's coming. SIF is a possibility for version 2.0

[maniac]

Those using CMIS, I've got a PHP solution which works, and extracts data from CMIS into a mySQL table, where I can manipulate it to do virtually anything I need.

 

At the moment it extracts Data set ID, Student ID, Forename, Surname, Year Group and their group memberships for all students currently on-roll. I'm in the process of wirting an interface to import this data into moodle, it's in its infantcy at the moment thou.

 

If anyone else wants to extract information like this from their CMIS database, get in touch, and I'll share the script with you. It's perfectly safe, as it only reads data from your database so it can't mess it up.

 

You will need some knowledge of mySQL and PHP to set it up, as there's no fancy install script or anything, but it's really quite simple.

 

Mike.

[webman]

Dammit. Why can't SIMS users have CMIS-style access to our own data?

[CyberNerd]

Dammit. Why can't SIMS users have CMIS-style access to our own data?

 

It sort of does - you can still access the SQL directly, we have some custom field in our SIMS and have used them to get moodle talking to a view of the SIMS database so moodle can pull out all the SIMS courses.

[webman]

@CyberNerd: Yes, that would work... if you want to void your support contract The "SDK" recommended by PhilNeal is far from adequate.

[johnymac]

You will be able to soon export from SIMS to import to LDAP automagically.

 

I will post my code online soon, we just had to add some code for dynamic exchange storage groups in multiple hosted solutions.

 

I'm hoping to finish up before xmas!

 

If anyone is realllly hungry I can email them my work to date however it may require some tweaking to work!

[limbo]

Those using CMIS, I've got a PHP solution which works, and extracts data from CMIS into a mySQL table, where I can manipulate it to do virtually anything I need.

 

At the moment it extracts Data set ID, Student ID, Forename, Surname, Year Group and their group memberships for all students currently on-roll. I'm in the process of wirting an interface to import this data into moodle, it's in its infantcy at the moment thou.

 

Wait until you get around to assessment data and lesson by lesson attendance - it is a whole lot more involved! You can get the data out, but doing it quickly is another matter (particularly if you have a lot of students with a lot of data!).

[StArbiter]

@alcinm I see you and Phil are competing for the most comprehensive posting awards !

I thought Capita went down the IMS route some four years ago in the absence of any real guidance from the Deaprtment or Becta. But it was not in isolation. They were in a working group with Granada, Microsoft and Capita that evaluated all the options including SIF and found for VLE integration that IMS was the best solution. All four then started developing to this solution and gave their feedback to Becta. Fronter also contribute deto the standard. It was only much later after this date that Becta announced SIF. All the credible demos I have seen by VLE suppliers on the Becta framework, including Netmedia and Fronter, have a IMS link. IMS is also the method used in our school for working with the SIMS learning platform. I am sure I have seen another research study that showed IMS was far superior to SIF and will try and dig it out - but it's no surprise that Becta found SIF superior.

 

The real important point you make is regarding the cost of SIF per transaction or in fact any cost for SIF implementation. I am not sure where your USA evidence comes from but in speaking to Pearsons who have implemented SIF in the states the cost was horrendous and they feel SIF could be a disaster for UK schools. You appear to work for a small innovative supplier and would do us all a favour if you could find out all the SIF costs and post them here. We already use VLE's from multiple suppliers and keeping them all fed is just about manageable. We really need your help to understand the real cost of SIF.

[torledo]

I have limited programming and minimal DBA exeprience. So, I've been reading various threads about extracting data from SIMS/CMIS and other proprietary systems from a newb perspective.

 

There's been a lot of talk about API's, XML and webservice, business objects (I'm assuming this refers to the business logic layer of th app) even reference to COM objects aswell as plain old SQL queries and crystal Reports when discussing the various ways of communicating with the CMIS/SIMS database. Could someone explain the limitations or issues with the various technologies discussed and how they relate to what Capita are or aren't doing. And the issues with access to the 'database' which I would have thought the school are responsible for maintaining while Capita as the developer would maintain the application code and business logic. Could someone please give a quick 101 for a non-SIMS/CMIS admin. I'm curious.

 

From my own point of view as a sysadmin what would be wrong with using perl scripts to extract and work on the data you need ? Is there a need for any kind of middleware at all ?