Ecclesbury Posted July 6, 2015 Posted July 6, 2015 Hi all, Has anyone had a chance yet to scrutinise the new Level 3 VA shadow data? I have had a quick glance and am now a bit perplexed! I've scanned the technical documents and as far as I understand it students with the same KS4 APS should have the same predicted score in a given subject. Equally, as APS rises, so should the predicted score. On the students level data that we have been sent it looks like the predicted scores bare little relation to the APS scores provided. Just taking Biology A2 as an example we've got students with higher APS but a predicted score that is 27 points lower. We've got students with identical APS scores but predicted scores that are 10 points apart. I've already emailed them to ask where I'm going wrong, but I'm sure there's someone out there that can shed some light on my error. Thanks.
MattMitchell Posted July 7, 2015 Author Posted July 7, 2015 I've not looked at ours thoroughly, but having had a look at Biology AS (All) our data shows the same kind of issues: 1 - students with the same prior attainment, yet different estimate values 2 - students with higher prior attainment, yet lower estimate values than their counterparts with lower prior attainment (2) shouldn't happen if they're using a monotonic (steadily-increasing) model, which is the whole point of their using a pool adjacent violators algorithm (1) also shouldn't happen, as it suggests a non-deterministic model My gut feeling, is that they're either a) modeling based on the old APS for KS4, but displaying the new APS for KS4 on the sheet b) modeling based on the new APS for KS4, but displaying the old APS for KS4 on the sheet I'm planning on matching it back against the old-style L3VA data, so have a look at the differences, although it seems like a waste of time if the model is broken! I suppose it's worth trying to work out why everything looks wrong at least
Ecclesbury Posted July 15, 2015 Posted July 15, 2015 I've not had a response from the DfE yet about this, but I did get to spend a few more minutes looking at it. It seems that the predicted outcome is affected by the actual outcome. So 2 student with the same APS and the same A2 grade will have the same predicted score (and hence the same VA.) However, two students with the same APS but different A2 results will have very different predictions. I'll be completely honest, in the last week of a very long term...and a teething baby at home...I don't have the energy to wrap my brain around this. I feel like the answer is within my grasp though. Can anyone suggest why a value added model would adjust predictions based on actual outcomes. A ghost of a memory at the back of my mind makes me think they did a similar, but smaller scale, thing with the Progress 8 shadow data.
MattMitchell Posted July 15, 2015 Author Posted July 15, 2015 Hmm, I didn't think it was meant to be a multilevel model, and even if it were, you wouldn't expect a student-level term in it. Can't look at this till over the summer now, but it's going to bug me I think!
MattMitchell Posted July 31, 2015 Author Posted July 31, 2015 Well, well, well! They've released an updated version of the data, where, although the headline measures haven't changed, the estimates seem to be deterministic; i.e. same qualification/subject, same prior attainment, == same estimate. I have fifteen cases where more than one student has the same prior attainment in the same subject/qual, and the data all stands up now. I've also checked, and there are no situations where a higher prior attainment results in a lower estimate. There is a small number where a slightly higher KS4 att gives the same estimate, but that's not inconsistent with the model. So... The big question is: is there a way we can compile all this data, without breaching confidentiality on the student side (I'm thinking qual code, prior att and estimate only), and without breaching Crown Copyright? Can it be done? Or do we need to FOI this?
Recommended Posts