Going to meet 'The Man' on your behalf

[Dos_Box]

Tony (Grumbledook) and I have been invited to a meeting with Ofqual (Ofqual - Welcome to Ofqual) in Coventry to discuss the issues around online testing. We are not sure exactly what format this meeting is to have (there are to be many relevant parties present), but we have been asked to represent the technical side of educational IT and put your views forwards, so we shall keep you posted on what happended once I get back in the EduGeek office on Tuesday. It may be they don't want a repeat of the KS3 testing debacle!

Watch this space

[witch]

Ohhh - you have reached exalted heights!

Best suit, of course!

[Dos_Box]

Ohhh - you have reached exalted heights!

Best suit, of course!

 

Just back from the cleaners

[webman]

If you're taking ideas with you:

 

We would greatly appreciate web-based testing with minimal-to-zero client-side software installation. This ticks boxes for a) mostly cross-platform and b) gives us fewer headaches.

[budgester]

If you're taking ideas with you:

 

We would greatly appreciate web-based testing with minimal-to-zero client-side software installation. This ticks boxes for a) mostly cross-platform and b) gives us fewer headaches.

 

I would agree with this, web based please. No client etc.

 

Username/password sync with our AD or Shibboleth implementation.

 

With data that will import straight back into our MIS systems without too much faffing around.

[alan-d]

An option to pick the levels expected from the students.

 

The reason we kicked out the KS3 system was because it was far too easy for our high acheiving students. This meant it was basically a waste of time for assessment purposes, especially as extra time had to be spent showing the students around the system etc.

 

Technically we had no real problems with the system it just didn't reflect what the students used on a daily basis and levels were set too low.

 

Oh - and another vote for Webmans suggestions

Edited November 20, 2009 by alan-d

[srochford]

Web based generally a good idea

Flash/shockwave OK but watch out for visually impaired students

System must save state at the end of each question - this means that if a machine fails; user closes browser; world ends etc nothing is lost

System must allow student to abandon test for whatever reason (time runs out, feels ill etc) and what has been done so far is marked (just like paper test)

Should be mock/practice tests available in same format (very disconcerting if they come into a test and it's completely different from anything they saw before)

[pete]

Dependencies on outdated versions of libraries is my biggest annoyance.

 

Keep the technical documentation up-to-date. If you have existing common issues with the software that you've just sent us, stick it in the accompanying (.pdf) manual - don't make me search for something you knew about and could have provided up-front.

 

Examples of good online testing/survey software that always works, always has sufficient capacity (load balancing at their end):

 

Midyis & Alis from Intuproject.org and (recently) the ECM Perceptions Survey.

[GrumbleDook]

One thing that comes up is software that needs particular ports opening and the problems some schools have going through this ... how much of a problem is this now? Configuration to go through your RBC's proxy or firewall? Any real world experiences about how well or poor this is for some schools would be appreciated.

[Busybub]

Username/password sync with our AD or Shibboleth implementation.

 

No thanks, in the past our kids have routinely swapped their passwords with their mates leading to all sort of problems, really don't want them logging on as somebody else!

 

Whoever sets up the students on the system needs a one click solution to randomly generate an access code for each test taken by each student, which can be distributed at the start of each test. The ePIPS system works well in this regard - students access the test software, enter their assigned code, and it launches the required test matched to their name.

Edited November 20, 2009 by Busybub

[Busybub]

System must save state at the end of each question - this means that if a machine fails; user closes browser; world ends etc nothing is lost

System must allow student to abandon test for whatever reason (time runs out, feels ill etc) and what has been done so far is marked (just like paper test)

 

Absolutely!

 

Lost count of the number of times the KS3 test has crashed out, been competely unrecoverable, leaving the kids with no marks at all. Also with questions not appearing in Midyis/Yellis CAB tests. There is then the hassle of rearranging to take the test which could be an issue in itself if our facilities operate near capacity.

[Busybub]

Teachers need more feedback. I remember our last ICT teacher moaning a great deal about the levels given for the KS3 test, and no other information about which questions the students answered incorrectly.

 

It would help to have some facility where the test administrator could log on and have access to the answers that the students supplied, so that teachers can assess where they went wrong.

[tom_newton]

One thing that comes up is software that needs particular ports opening and the problems some schools have going through this ... how much of a problem is this now? Configuration to go through your RBC's proxy or firewall? Any real world experiences about how well or poor this is for some schools would be appreciated.

 

We would welcome any info ahead of time about what schools might be needing, and would happily test things and make any changes to our rules prior to going live. Feel free to point any web-based testing personages at me for such cooperation

[GrumbleDook]

We would welcome any info ahead of time about what schools might be needing, and would happily test things and make any changes to our rules prior to going live. Feel free to point any web-based testing personages at me for such cooperation

 

Personally I would say it shold be expected that any online testing should be tested to work via *any* Becta accredited filter or accredited filtered connection. That could be a point for me to take along. how the various companies do that will be interesting.

[srochford]

Personally I would say it shold be expected that any online testing should be tested to work via *any* Becta accredited filter or accredited filtered connection.

 

forgot earlier - it must work with ISA server (and probably Squid?) - too many of the suppliers seem to have any idea that lots of schools don't have direct internet access but go via some kind of filter or proxy or both!

[tom_newton]

@grumbledook - yes - a good point. BECTA are trying to kick off regular (if not frequent) meetings for filter vendors. Would be nice to make sure that those of us who do give a sh1t about education get the info we need to keep our products working as they should.

[john]

Simple and idiot proof, don't take the Edexcel Promisser any futher that is a joke and a big waste of my time. Any online testing software needs to be completly web based and just need a normal internet connection on port 80 thats it! Nothing more complex than that

[Guest Guest]

The problem with no client is you cant lock the user down; ie you cant stop them using a calculator or a spell checker.

 

What if the "client" wasnt a client in the true sense but just a program which can lock the user into a single web browser session. That way you would only need to create a client for each platform, and in theory (if you get it right) you should never need to do any other client side updates. Any update to the exams would be purerly server side.

[john]

I'd far rather make a mandatory profile that launches IE on a specific home page as the shell of windows and lock off calc, word etc than mess with Promiser.

[mortstar]

Simple and idiot proof, don't take the Edexcel Promisser any futher that is a joke and a big waste of my time.......

 

+1,000,000

 

If the representative from Edexcel offers this as a possible move forward please let them know that it is flakey as hell.

[Dos_Box]

Right, I'm back and can let you all know what happened and what it was all about.

Firstly. It was attended by pretty much every organisation and company with an interest and involvement with online testing and e-assessment (almost the same thing!). This included BECTA, QCA, AQA, the Exam Invigilators Association (I think they were called, his card is still in my jacket) and even the English Secondary Students Association. In total there about 30 organisations represented.

Secondly it was not about individual products and picking faults with them. This was about the BIG picture, about what we all felt was required to make online testing and e-assessment work in UK schools, and EduGeek had a very large input into the day, with Tony once again proving that there are few people in the UK as knowledgeable in the field of educational IT, policy and structures. I'm not sure if there will be any follow on meetings, but Ofqual certainly got a lot of feedback from us all, and enough to start generating some kinds of policies to ensure that testing can take place successfully across all UK schools. I'll list a few point we and others brought to the fore which you can all discuss if you like, particularly one that I shall highlight:

 

1. Standardisation - All testing platforms should operate over common ports and standards, and all RBCs and LEAs should be made aware of all servers and requirements so as to ensure that both developers and schools can run tests on systems which are already 'cleared'. This includes non-standard ports and server name lists so as to avoid accidental black listing, One company suggested that a list be submitted to BECTA every September so as to allow for development and upgrades to take place and this information can then be disseminated to the relevant RBC\LEAs.

 

2. Logistics. To run mass online testing requires a LOT of computers to be made available in manageable areas. The average secondary school will have 200 pupils in year 11. If they were all to be doing an online test at the same time it would create a lot of problems getting bums on seats in front of computers. Of course, one of the aims of online testing is that they don't all have to do the exam at the same time and the exam can be staggered. This problem however is magnified with the advent of BSF schools and the decentralisation of IT equipment i.e. no large ICT suites. In this event it is highly recommended that schools still in the planning stages of BSF ensure that a room can be cabled up and have enough desking to be converted to an exam suite in the future, because online testing IS coming, make no mistakes about that, and you need to plan your resources appropriately.

*Please discuss this as some parties wanted feedback about just how a current school could provision for a mass online exam if they had too.

 

3. Training. Remember the KS3 testing platform any of us had to install and get working then ensure that all the different parts works as they should? Remember that we no training to do any of the things we were managing? Well, this should change. Neither techies, invigilators or school admin staff has any training in the testing systems, yet all were involved to one degree or another. The training mostly went to teachers who never used the platform. This is a mis-direction of resources, and hopefully in the future anyone who has need of training on a particular exam system should be trained correctly, and simply because they are a teacher.

 

There are a few more points, but I shall post up the report on the day once I receive it, but in short, EduGeek put your views across very well and they were all noted and well received. Now all we have to do is find out how to test 200 pupils at once!

[alan-d]

Testing 200 pupils at once.

 

Top of my head response;

 

Normal ICT suites may not be aqequate as very often the students can view each others screens.

 

Laptops in a hall - wireless connection for 200 connections simultaniously. Possibly hard wire them temporarily but still leaves you to find 200 laptops from around the school which also has a knock on effect for those lessons that would normally be using them.

 

Possible solution - the exams come on a numbered flash drive and are completed then handed in like exam papers are today. Sent to the exam board and uploaded.

 

The exams vary to enable 2 or 3 sessions. This would mean that those students taking the exam in session 1 won't be able to tell those in subsequent sessions what the questions are.

 

Brain dead at the mo - may come back with more later

[Dos_Box]

Not beingSecondary I'm not to clued up on this so please ignor if I'm way off but how about the exam boards supplying the kit needed? Pre-configured server in a box complete with netbooks for example?

 

Not sure, this was beyond the remit of the day, athough I did talk to many exam board people and they were suggesting it would be 'live' over the internet for many tests, all done in real time and 'on demand' i.e pupils took exams when they were ready too directly linking in with the boards servers and the results would be published directly to the school and pupil in a very short space of time.

There will of course be 'local' models which they sent the finished 'papers' to to a central source to be marked.

[Harlquinth]

The average secondary school will have 200 pupils in year 11.

.

.

.

There are a few more points, but I shall post up the report on the day once I receive it, but in short, EduGeek put your views across very well and they were all noted and well received. Now all we have to do is find out how to test 200 pupils at once!

 

We have approx 150 pupils per year here - we have four computer rooms with x32 computers in each, and one computer room with x24, so assuming that the rooms could be freed up and enough invigilators found then there might just be enough computers, however I'm sure I remember seeing some guidelines somewhere saying that only every-other computer should be used to prevent the students seeing each others screens, so that would cut down the number of usable computers a lot.

 

Peter.

[GrumbleDook]

A slight addition to this thread ... feel free to have a look at this questionnaire (and the website it is asking you to comment on) and please complete it ... also pass it on to your Exams Officer and whichever member of SLT deals with curriculum development (ie choosing which courses / exam boards your school uses).

 

Questionnaire : Efutures evaluation