Head of of Ofqaul asks: why aren't students using computers in exams?

[psydii]

BBC Radio4 Today Programme 25/02/2011 0732

Should computers replace pen and paper in all exams? Isabel Nisbet, outgoing chief the Ofqual qualifications watchdog, argues that GCSEs and A-levels will become "invalid" for digitally native pupils if writing materials are retained. She debates her proposal with Dr Sheila Lawlor, director of centre-right think-tank Politeia.

 

BBC News - Today - Today: Friday 25th February

 

I'm quite excited to hear this debate out in the open. Having used a PC in my own GCSE's many many years ago, and run GCSE's in a SEN school with 60+students all sitting their public exams on laptops, 9 years ago I am continually disappointed by the lack of progress in this arena.

 

In my experience it is easy and cheap, assuming your organisation has invested in appropriate technology already. However 60 is not the 300 that I'd need to do the same thing in a large secondary today.

 

The security fears she raises are easily addressed. In fact that this link here is Using Windows 7 and UMRA for Controlled Assessment almost exactly as I did it 9 years ago.

 

The question is how do we get enough devices to pull it off? Or is it better to use the controlled assessment model and abandon the idea of terminal examinations simply because they cannot be adequately assessed in todays' environment?

[beeswax]

Just caught the end of this as Ms. Nisbet said she was going to Singapore and a little bit about Chinese calligraphy. I'll try and catch up with this when I take a break from decorating. I've a soft spot for Ms. Lawlor, it's the Great Grimpen Mire.

[pete]

In my experience it is easy and cheap, assuming your organisation has invested in appropriate technology already. However 60 is not the 300 that I'd need to do the same thing in a large secondary today.

 

It's logistics. All of our (~880) kids do exams in the summer and GCSE and AS/A2 mocks are done in February. Assume each A-level kid (~280) is doing at 4 exams of at least an hour each and GCSE (~600) are doing 9-10 exams of at least an hour each. This discounts multi-day practicals (Art, Food tech, Chemistry etc).

 

We have 3 halls, which can seat 200, 250, 100. So 550 kids can be doing exams at once and we need 3-6 invigilators. We'll also have smaller exams going on in other rooms.

 

Then consider our IT setup is based around the 30-34 kids in a classroom unit. Even with all the computer rooms full, there's no practical way we can run kids through exams at the same speed. Plus trying to find reliable invigilators who can say "yeah, you've got caps-lock on" without calling IT support is trickier (in this neck of the woods) than you might think.

 

For 4 weeks in a year, it's crazy exam season. The rest of the year, there's no call to have hundreds of kids using computers in such a synchronised activity. This is before we start to consider that some lessons are being taught during exams and hijacking resources from them would be rather unpopular.

 

The logistics solution? Each kid has an exam-suitable device that they use as their primary computer in school. During exam season these devices boot into $other_os or (insert method of ensuring no access to local devices/data/internet) and load the exams from an in-house server via (decent, properly scaled, managed) wireless. Since each kid has one stashed in their schoolbag, it doesn't matter where they're taking the exam.

[irsprint84]

Xen client could do this if you had enough RAM

[Domino]

Xen client could do this if you had enough RAM

 

You'd still need physical hardware for each student though.....

[srochford]

One of the things that Isabel Nisbet said in the interview was along the lines of "this has always been considered too difficult - I'm now just taking it off the 'too difficult' pile" to get people thinking about it" and I think that's fair.

 

There are loads of reasons why this could be a good idea and loads of potential problems. We're using it for a limited number of exams (mainly where you're being tested in things like programming or where you need to use a particular piece of software as a rather special calculator)

 

I'd guess that in schools it would make sense to start by using computers for exams where they provide a real benefit in the same way that calculators started being used in exams because it meant that you didn't have to make the arithmetic easy and could concentrate on the physics, chemistry etc.

 

It isn't going to happen this year but I'm sure it can happen. Many schools have a good framework in place (eg the facility to lock down machines, reset machines to a known state at the end of a day, restrict internet access by user group or machine group) and so on but getting enough hardware will be a challenge. Is this the time when we change away from everyone doing the same exams at exactly the same time?

[joe90bass]

...but getting enough hardware will be a challenge. Is this the time when we change away from everyone doing the same exams at exactly the same time?

 

I can't see it being done any other way, a typical class of 30 PCs has the PCs too close together to use all 30 under exam conditions, unless you were to put a partition between them....

[Jamman960]

What if the exams were based on a bank of 1000+ questions? then there would be less of a need for the entire country to sit the exams at exactly the same time and would allow for smaller groups to sit the exams at different times

 

I can't see it working at all without changes to the exam format.

[joe90bass]

What if the exams were based on a bank of 1000+ questions? then there would be less of a need for the entire country to sit the exams at exactly the same time and would allow for smaller groups to sit the exams at different times

 

I can't see it working at all without changes to the exam format.

 

I started typing something along those lines and then hit some random key combination that sent IE back a page.....

 

My thought on that was that some exams build on the previous question, or have a flow to them so it wouldn't work in all exams, but for those exams that could be done this way it would allow more PCs to be used in a room.

 

But as you say, there needs to be changes to the exam format, it isn't just down to a technological response.

[CyberNerd]

annoyed about this. IMHO it is mostly about exam boards wanting to pass their costs of marking etc onto the school. ie - the schools pay for equipment based on the exam boards criterion which is contained in the "Instructions for the conduct of the examination" (ICE) document. The ICE document rarely pays attention to the pedagogic setup of the IT suite, and focuses entirely on the exam board saving money. With the already high costs of the outsourced exam boards, they should stump up the costs of their examinations!

[plexer]

I was talking about this at bett, I think it needs to be effectively a different test for each student and also they need to be allowed to take the exam when they are ready to take it.

 

Ben

[srochford]

The "different test for each student" is already there in some of the tests - the Promissor stuff was doing this years ago so the exam seating model becomes much less important.

 

There are some exam boards which allow exams to be taken on demand - not sure if this is true for any UK qualifications but it certainly happens for some of the Cambridge International stuff (I used to set the papers!). The issue with this in many schools is that the exams end the year and nothing happens after the student has done the exam but you could have a different system.

 

It's not just about passing costs on to schools (although I'm sure there's an element of that in there) but making exams fit in better with the teaching and learning. It's not going to happen tomorrow but we can start thinking about how we would do it and how we make sure that the system develops in a way which doesn't cause unnecessary expense, disruption etc etc.

[CyberNerd]

I think if ofqual really want to go down this route then they should standardise an online testing process or online exam centre. many exam boards with their own software just makes a mess. (sometimes incompatible with each other, IIRC there was an exam board a couple of years back that required IE6 for one piece of coursework and IE7 for another..)

[GrumbleDook]

This really isn't that new in their discussions though ... http://www.edugeek.net/forums/general-chat/45189-going-meet-man-your-behalf.html saw me and Chris go to talk with many others about it and the biggest barrier is how the curriculum is set up in the first place of only testing at the end of defined periods, not when students are ready.

 

We are back to discussing the KS3 ICT tests again ...

[SteveBentley]

The key part of this is the comment from the OCR

However, a spokesman for OCR said the board's focus "was not to make existing paper-and-pen tests electronic but to explore ways that computers can add real value to assessment".

BBC News - Ofqual head: End paper exams for digital generation

 

And this is where the nail is hit firmly on the head. The real question should be "why are students still doing (so many) exams", not how can we transfer exams to a computer.

 

The main reason for persisting with exams is a paranoia about plagiarism - and an invalid belief that exams are the only form of assessment which can't be plagiarised - and a sense of that's how it's always been done so it must be right.

 

But exams are flawed because they take place in an artificial setting, take no account of differing learning styles. It's very much open to debate whether exams actually assess anything useful.

[alan-d]

The KS3 ICT tests mentioned by GD above were sound in theory but the actual implementation was flawed in many areas. The software was sent out without proper testing and lots of schools had problems installing and running it. We were one of the lucky ones who ran the sofware without problems but found that a lot of time was spent teaching the staff and students how to use it which took up valuable teaching time. Our students are bright and picked it up quickly but even then it used up 1 or 2 lessons.

 

Ofqual need to set a standard that all exam boards should follow to allow schools to install and run examination programs that work on all operating systems from Win 2000 to the current version. They should also be realistic about what should be available during the exam such as internet access or spell checking facilities to allow for those schools that have very little in the way of IT support or have outsourced their IT Support. Guidance should also cover the procedures to follow in the event of equipment failure during an exam. The KS3 ICT Test was good at this in part but was useless if the fault was a hard drive failure. The current example of Ofqual and exam boards not setting a global standard and providing the means to carry out an exam process is the Controlled Assessments, just look at the posts on edugeek to see how many different ways that IT support have had to try and interpret the requirements and the man\woman hours used to provide a bespoke solution.

 

I think the future should be a locked account which has only access to an exam board web site to which the student logs into to carry out the exam. This should provide the security required as there will be no paper copies, the exam board has the results as soon as the exam is completed which in turn means fewer exam papers getting lost in the post etc etc etc.

 

As for computer suites being unsuitable due to layouts I would suugest fitting Anti glare\privacy screens to monitors during the exam season.

[john]

The major issue currently (as AQA are pushing for on-screen tests in some GCSE and A Levels and have marketed all subject leaders, SLT and Exams Officers on the mailing lists just before half term on it) is the JCQ requirements for the tests. How many of us have a minimum of 1 meter clear space between each and every screen in our IT rooms? I bet the bulk of us don't, the requirements are fairly impossible to meet in most peoples setups I suspect, without the making of a large number of custom exam screens to make each PC area into a cubical like you get in driving test centres etc....

[Chris_Cook]

 

And this is where the nail is hit firmly on the head. The real question should be "why are students still doing (so many) exams", not how can we transfer exams to a computer.

 

 

Wot he said. Students should be learning stuff in school, not constantly being graded on stuff that they haven't had a chance to learn yet.

[GrumbleDook]

And we get back to the question of "Does the pig get any heavier if you constantly weigh it?"

[alan-d]

And we get back to the question of "Does the pig get any heavier if you constantly weigh it?"

 

If you feed it enough scraps it will

[witch]

Having watched three children go through the GCSE and A Level process - I really think they need to do something. My children are used to working in a different way, essay-wise than the way you work if you hand-write an essay. Wheras, when writing by hand, you have to get it all in the right order pretty much on the first go, when you write on a computer, you can just put down the information and then go back and get it into some sensible order. This is how they do it on an every day basis which makes exams quite stressful and none of them believe that they do as well as they would if they could do it the way they have all the way through school, on a computer.

It is the editing facility that is missing - it isnt that they can't write a coherent essay, just that they do not do it that way and therefore are not practised at doing so.

There must be an answer - slates or something similar?

[pete]

I think the future should be a locked account which has only access to an exam board web site to which the student logs into to carry out the exam.

 

Given how unreliable their sites are for general use, is it sensible to trust exam boards to invest in the infrastructure to ensure they've sufficent capacity and load-balancing to handle the exam season? Bearing in mind it'd be every written exam for that board.

Edited February 28, 2011 by pete

[wesleyw]

@GD: Depends if we're measuring at a quantum scale or not

[GNewnham]

The new JCQ instructions for conducting examinations state 'Centres are allowed to provide a word processor to a candidate where it is their normal way of

working, unless an awarding body’s specification says otherwise.'

 

ie any pupil may use a word processor if it is their 'normal way of working'. This change has come in this year.

[alan-d]

Given how unreliable their sites are for general use, is it sensible to trust exam boards to invest in the infrastructure to ensure they've sufficent capacity and load-balancing to handle the exam season? Bearing in mind it'd be every written exam for that board.

 

I don't see why they couldn't invest in improving the infrasrtucture, they would reap most of the cost back by not having to produce paper copies and by cutting down on manual marking. At the moment they expect us to overcome the same difficulties without any additional income or resources.