LosOjos Posted October 22, 2014 Posted October 22, 2014 I've just "finished" (it will change, I have no doubt about that!) my first lot of Progress 8 based analysis. It's a major change and it's meaning we're having to really change the way we look at data. As it's such a new thing, I'm wondering what methods other school have gone with to analyze this? For each subject, I have calculated an expected average grade based on the expects Attainment 8 score for all the pupils taking that subject, plus the current estimated average grade (all 'fine' grades, i.e. +/=/-). We're basing this on 2013 estimates for now as this is the latest we can currently find from the DfE. In the past, we've always provided % on/under target for subjects and initially we tried to do the same with P8 by rounding the expected P8 to the nearest grade and comparing our teacher's estimates to that. This highlighted a problem though - if you try targeting like that, because of the range of points assigned by the expected P8 table you could potentially end up with 100% "on target" but a VA of -0.9. This is a problem. We also looked closely at Maths and English and noticed another issue raised by P8 - three levels of progress is not necessarily good enough any more. For instance, a student coming in to school with a KS2 fine level of 5.2 (5c) would need to achieve at least a grade B at KS4 to have a positive P8 VA - which is fine, 3 LOP as we're used to. However, a student at just a 5.6 (middle of a 5b) would need an A to have a positive VA - 4 LOP. This raises questions as to how English and Maths will be judged in future, as clearly if we're targeting for 3 LOP, we could potentially be dooming the child to a negative P8 VA whilst our analysis looks as though they're on target. So what I've now decided to do is scrap on/under target measures and provide just a VA for each subject/student. This is better because it doesn't give the false impression of adequate progress, but frustrating for staff because they might have a kid estimated for a C, making 3 LOP, but with a negative VA, and I know that getting these students to a C was hard work, a B almost impossible! On the flip side, the lower end of the table (level 1s and 2s) have such low expected P8 scores that most of ours have incredibly high VA figures. As a school we look set for a positive VA, which is great, but how we monitor this at a subject level I'm struggling; if a students expected VA is 42/4.2/D-, who do you put the pressure on to get that student a C to ensure a positive VA?
LosOjos Posted October 22, 2014 Posted October 22, 2014 Missed my main question! What are you all doing in terms of P8 analysis?
mark80 Posted October 22, 2014 Posted October 22, 2014 In short we are using SISRA However i have also pointed out the shortfall of everyone hitting their "target" and still potentially getting a -VA. We have done a broad stroke anyone on 5c or less is 3LoP and 5b and above to 4 LoP, with these in SISRA as Targets i can look at the Att8 & Prog8 in more detail. You know as well as i do that students do not fit a formula and trying to be all things to all men is near impossible. LoP, VA, Att8, Prog8, residuals are, in my mind, not to be used as a sole representation of a student but used in conjunction with all available data to paint a picture. Sadly though teachers are now forced to commit to a target for performance related pay purposes and they will be told by their SLT to be accoutable to just one of those measures. These are also easier for OFSTED to anaylse.
mark80 Posted October 22, 2014 Posted October 22, 2014 Just looked at SISRA and if everyone of our students hit their targets based on 3/4LoP (non Eng&Math are based on Average KS2), we would be Average Total Attainment 8 51.06 Average Total Attainment 8 (A*=58 Points) 404.07 Average Total Progress 8 0.30 Progress 8 Upper Confidence Interval 0.45 Progress 8 Lower Confidence Interval 0.14 What is worrying though is that even if all our students hit the 3/4 LoP in Maths & English our capped 8 VA is still 982, with confidence intervals of 991 & 974, making us Sig- than the national average! So we have a Sig+ Prog/Att8 and with the same data have a capped 8 Sig-
LosOjos Posted October 22, 2014 Posted October 22, 2014 Yeah totally with you RE no one data set being the whole but, as you said, with performance related pay we have to provide a comparable benchmark. I think what worries me about Prog8 is it's so vague, we have no history, no contextual data, NOTHING by way of comparison other than taking the DfE's word that their estimates represent national average - a 'fact' we know is as good as false as it's based on incomparable retrospective data! The figure alone just doesn't seem to mean much. I mean yeah, OK, it's supposed to show you where your school sits in term of national average and yes, you have that -0.5 floor standard, but how do you translate that to relevant areas of intervention? The only way to guarantee a positive VA is to make all subjects aim for a grade higher than Prog8 expected, when they're already struggling in some cases to meet it. Perhaps it's a problem for our school, but I suspect many will be in the same boat. We're just lucky we have such a high % of low attainers - they smash their P8 expected!
NormalDistribution Posted October 24, 2014 Author Posted October 24, 2014 As alluded to, it's pie in the sky, because the national attainment 8 estimates that are provided, and i assume are used as the basis of P8 modelling in SISRA are almost certainly under-estimates particularly for lower ability pupils as currently lower ability pupils are less likely to be on pathways that give them 3 ebacc subjects, but come 2016 this will have changed. In turn this will change the estimates and the P8 outcomes by quite a bit i imagine. I started a P8 tool last year that allows you to model different national scenarios, but again I'm only really using this as a best case and worst case scenario. have a look if you like: https://drive.google.com/file/d/0B8cR7OtZSTKlUXBqYmM5NHNTOVk/edit?usp=sharing
LosOjos Posted October 24, 2014 Posted October 24, 2014 @NormalDistribution - thanks for sharing. Looks a lot like what we're doing (unfortunately can't share in it's current form, but will try to get round to anonymizing it if any one interested). Like the idea of your 'What If' scenarios though - will look to adapting that
LosOjos Posted October 24, 2014 Posted October 24, 2014 How are you estimating at all baskets being filled? Taking the mean to populate empty slots, or something else?
NormalDistribution Posted October 24, 2014 Author Posted October 24, 2014 yep so it assumes that if they filled the empty slots they would achieve their average grade
mark80 Posted October 28, 2014 Posted October 28, 2014 Great stuff! i see you are using the Average KS2 as a baseline for all subjects, is this how it will be measured for English & Maths or is this just the starting point of the Progress 8 expected achievement
NormalDistribution Posted October 29, 2014 Author Posted October 29, 2014 That's how it will be measured until the KS2 scores change, although it will use fine grade and not sub-levels but I built the tool on sub-levels as that what we use currently.
Recommended Posts