MIS Performance for generating large (300K+) row Reports

[RootsSchImp]

Hi,

 

Does anyone regularly pull out really big reports from their MIS - I'm thinking specifically along the lines of a results backup of every assessment result for every student. I've done this recently for a SIMS school to make up a back up archive which came through eventually after about half an hour. I've tried it on a cloud system (Bromcom) and not been able to pull the data through because it affected other processes and had to be killed.

 

If you generate this kind of big report (300k+ rows) what system do you use, how long do you expect to wait and will it affect other users?

 

Cheers

 

Ed

[Steve21]

That isn't really the kind of thing we'd ever use as a backup so never run one like that. Any reason you aren't just doing a normal backup etc?

 

If you're doing anything that effects every member of any MIS it's going to grind to a halt unless you've given it huge specs to run a single report

 

Steve

[mukz]

This reminds me of the time we caused a Pearson e1 outage because of the reports that we were trying to run.

 

Now that i look back at it, i don’t really know what the point of it was!

[Michael]

I have to agree with others - MIS system or not, querying any database system for 300,000 entries is abnormally large.

 

Chances are, a SIMS backup is run every evening and all cloud MIS providers (to the best of my knowledge) perform backups their end too.

[RootsSchImp]

I was cleaning a Sims system of old aspects and result sets. The school wanted to be able to refer to the old results without having to restore everything to Sims. I've also come across solar reports (maybe not quite as large) for schools that export tracking data to excel for analysis.

[Steve21]

In that example though, you just run it once out of term time and it's done No need to regularly run it. Same as things like bumping up new years, changing mass groups etc

 

Steve

[RootsSchImp]

That's true, although it seemed like it was right on the edge of what the server could handle which surprised me and I wanted to get a sense of how normal that was/whether anyone else had issues running their biggest reports.

[Michael]

I'd say breaking it down per school would be more practical. Even better you could schedule it out of hours to generate (where possible). Some systems offer such function and others don't.

 

20 x smaller reports in theory should run faster than 1 x massive report for example.

[matt40k]

I'd question your cloud MIS if it couldn't scale up. 300,000+ rows is alot but it's small fry in the real world.

[RootsSchImp]

I'd question your cloud MIS if it couldn't scale up. 300,000+ rows is alot but it's small fry in the real world.

 

I know right? Although @Michael makes good points about breaking it up and running the parts out of hours: so often more productive for the user to change than the machine.

[matt40k]

True, but hardware is cheap, humans aren't. Easier to hit the go faster button then re-work the report. Also map reduce anyone