MIS & Data Interoperability

[GREED]

I have been having an interesting debate with a fellow educationalist about what is important about data interoperability and the availability of tools to enable data to be easily accessable from an/any MIS.

 

There is no question that data availability inside and importantly outside the MIS is critical (My former employment taught me some lessons there). But how should that be accessed? Should this be through an open API? Through supplier tools? Through third party managed tools?

 

There are many interests to suppliers of third parties in particular, often the simplest solution from a schools perspective is not always the easiest, cheapest or best solution long term for software suppliers.

 

Indeed, does the concept of supplier built tools add an air of 'lock-in', does such a premise give the MIS supplier too much control, or allow them to manage the internal data processing and other overheads to an acceptable limit?

 

Any and all thought from all welcome.

 

(PS this is a general interest from a debate I am having, in no way is it related to my current work, so please do not assume this is advertising nor fishing)

Edited February 15, 2014 by GREED

[PhilNeal]

I can't imagine any MIS cloud supplier wanting a third party creating routes into the tables. The supplier is responsible for performance and will incur costs if inefficient code is accessing tables not to mention concerns over bypassing all validation checks.

[GREED]

How do you legislate for that though Phil? The performance aspects I mean

[localzuk]

The only realistically feasible way of ensuring data interoperability that I can think of is for the government to draw up a single API for all MIS providers to implement.

 

It will, of course, only ever be able to handle a subset of data as different solutions will innovate in different ways, so exporting that data for use in another system might be impossible etc...

 

The current solution for moving between MIS systems is woefully inadequate to say the least.

[GREED]

Data Exchange will provide and push part of this. But not MIS migration

[PhilNeal]

Government IT projects generally fail!

[GREED]

Do you believe DE will?

 

I cannot say I forsee it failing it is now so far into it would damage so much given a lot is waiting on completion. It's nice working on the inside for a little bit

[PhilNeal]

Let's just say its "courageous" to quote "Yes Minister"

[GREED]

Bit before my time....

[PhilNeal]

You should listen them on iPlayer. Nothing changes in politics!!

[matt40k]

Ah you guys, you do make me laugh. I fear Phil may be bang on the money, when I read you post about Data Exchange it did remind me of ContactPoint - a brilliant idea which died a death when someone else stepped into no10 and the public head was turned to another problem. Lets hope it does not endure the same fate.

 

How do you legislate for that though Phil? The performance aspects I mean

 

Way I would do it is to have core tables, then have a third party tables that you charge third parties to use, price would depend on things like column type. This would insure the single location for all MIS data, allow third parties to have enough control but not have to worry about a million and one things like security. They would of course still have to use the same "framework" to access the data.

 

accessing tables not to mention concerns over bypassing all validation checks.

 

You don't access tables anymore Phil, that's so last gen. You would use something like MVC then you could give third parties access to the "API" layer so it goes via the same logic as your UI.

[GREED]

I sorry... charge? And you talk about being so last gen?

 

You might want to do a little bit of research there (I have, you see ), the charging model is becoming a dying breed within the MIS community.

[GREED]

It is funny actually within the halls of the DfE building, ContactPoint is taboo and you get stares if you say it out loud!

 

This project too was started long ago, the essence of it goes back to the BECTA report which was last government. End of the day DE (well, the data warehouse part) is a long overdue replacement for COLLECT as well as incorporating about a dozen other interfaces and databases dating back nearly 20 years. Cancelling at stage will leave the department we virtually nothing, as well as leaving the UK in the comparative dark ages compared to the likes of the US and Australia who are also moving along similar lines when it comes to data exchanging with the government and each other (schools).

[matt40k]

I sorry... charge? And you talk about being so last gen?

 

I've yet to see a free solution mate. Everything I see costs. For example...

 

I can contact SIMS, get the business objects then import\export anything and everything. I do so, and I also write back to\from whatever system I have. Let's ignore the cost of writing the interface and any other costs I may or may not incur. Then have to pay for someone to maintain this interface, for the interface to be security check (PEN testing for example) then actual physical hw costs which alas I cannot escape.

 

My idea, you'd just be paying the storage cost to the MIS provider. Which regards of how you look at it, you'll be paying anyway as it costs to store data. Just my idea, you're refining the design and cutting out any extras that are not needed. You also get get the benefits of 1 system also, such as one login, but you wouldn't be limited to 1 software provider. Of course this would only work if the MIS provider that did this, kept to it's core features.

 

RE: DfE yer, I can image. The whole one warehouse idea really needs to happen and it's good to see the whole 1 domain thing coming together under gov.uk (type thing).

[GREED]

I'm talking MIS providers charging for their API or inbuilt interface. I think if you look around you will find there is no charge for many a provider now. Yes you have all the other charges in terms of development etc, but not paying just for the privilege...

 

Don't worry this is part of my questioning to put to MIS suppliers next month

Edited February 16, 2014 by GREED

[GREED]

There is also the issue of how do you control how much x third party thrashes the server or services if tapping directly into the database, regardless of they are core tables or not. What if x third party wants to poll every 10 seconds for the entire database which in a new cloud installation could be for an entire LA so 100000 kids full records...

[psydii]

There is also the issue of how do you control how much x third party thrashes the server or services if tapping directly into the database, regardless of they are core tables or not. What if x third party wants to poll every 10 seconds for the entire database which in a new cloud installation could be for an entire LA so 100000 kids full records...

 

It's not the third party's fault if calling the published API brings the database to its knees.

[GREED]

It's not the third party's fault if calling the published API brings the database to its knees.

 

What?! Yes it is if my suggested scenario is put into effect. I could write one that could flood Internet traffic effectively a ddos. sure I might think I need it for my own uses and having such a badly writen real time data transfer might look good, but how can that be the MIS fault? There has to be control from the MIS side for all involved, especially if it is envisioned that an API is to be open to all.

 

I'm not saying that design consideration should be put in to make the MIS resilient but there needs to be check in place.

Edited February 16, 2014 by GREED

[psydii]

No, it is the MIS developer's fault if the API allows the third party to kill the DB performance.

 

Should you be able to ask for a list of all student marks per class from the student, course, and grade book tables? Yes.

 

However next time you want it that day you the (API user) should be asking only for "what's changed since I last asked"

 

The api behaviour should enable and encourage that, and discourage poor behaviour of the consumer (by weighting the priority of repeated request for full dumps such that day to day operations have precedence, for example).

 

 

Furthermore the MIS provider will be able to see what the 3rd parties are asking for and optimise database (tables, stored procedures, indexes etc) to maximise performance for both their customers and their 3rd party API consumers.

Edited February 16, 2014 by psydii

[GREED]

Coulda Woulda Shoulda... It should only ask for but doesn't guarantee a cheap app won't be poorly built.

 

So you are back into the old territory of having an MIS validate every integration, which of course there is a cost for (so third parties loosing out), then there is an overhead with constantly changing and managing the API to suit all, likely passed back to the third party, and then there will inevitably be losers when things need to be changed to balance performance and availability.

 

Again, it can and even would ENCOURAGE good behaviour... doesn't mean it will enforce it.

 

As an MIS supplier my first and foremost priority is to provide a good service to my customers with my software.

 

If too you have read that there is a changing ideal in the way providers work their API/integration model, then the procedures you have described do not fit, or mean a waking great loss for the MIS company if not passing costs onto third parties (which is the current model with a few supplier, and is crippling to some thrid parties and is actively discouraging innovation and choice in the market... not my opinion but the opinion of several I spoke too at BETT).

 

Is good to hear different arguments on this topic, exactly what I was looking for!

[psydii]

I'm advocating the MIS providers get into the 21st century and build 'web scale' api's to the enable their customers to pick best of breed applications to meet their needs.

 

What I see is MIS providers hunkering down in the 1990's model and trying to shifting the balance of power towards themselves and away from the customer.

[psydii]

Also I know at least one MIS provider who has an API (they need it as a marketing tool) but 3rd party will not build against it because the potential customer base is currently too small, even when the school has offered a months worth of developer-sallery scale money to get it done.

[Roberto]

I can't imagine any MIS cloud supplier wanting a third party creating routes into the tables. The supplier is responsible for performance and will incur costs if inefficient code is accessing tables not to mention concerns over bypassing all validation checks.

 

In both cases that's a decision for the customer to take - once fully warned of the risks of course. You presumably generate a SLA now that already only warranties performance and / or data integrity only while the system is being used in an 'approved' manner.

 

Lets be honest here, a "cloud MIS system" is essentially a hosted SQL database and front end. There's not that much secret sauce in that, and neither is it very difficult for the provider to monitor access and highlight anything that's being done in a 'non-approved' manner that is causing problems.

Edited February 16, 2014 by Roberto

[PhilNeal]

@Roberto - not really many cloud MIS solutions are multi-tenanted i.e. multiple schools in one database. (remember I am not talking SIMS - we are not in the cloud)

[GREED]

Someone is... well, was... well still is but not here....