FFT Aspire Progress 8 & DFE Progress 8

[mark80]

Morning

 

My local authority have recently sent all schools FFT Aspire documentation that calculates our schools Progress 8 measure, however it appears to be wrong. It claims at the top of the report to be "based upon unvalidated data (October 2014)", which to my knowledge is the same caveat that the DFE Progress 8 analysis tables as discussed here http://www.edugeek.net/groups/data-managers/563-progress-8-shadow-tables/

 

We currently use SISRA and using the raw data regarding progress 8 found here https://tableschecking.education.gov.uk/Web14/KS4Progress8Optin/FSG.User/Login.aspx both our SISRA and the DFE ReadyRecknor come out with the same value.

 

FFT are quoting our Progress 8 as -0.41 while DFE have set it at -0.34, I know the differences are small but as you all know the implications are massive

[LosOjos]

First thing that pops to mind is how are they calculating KS2 fine levels? FFT calculate these slightly differently to the DfE, so if they're using their own calculations, that could explain the difference.

 

An analysis of Key Stage 2 reliability and validity

[mark80]

@LosOjos thank you for that.

 

I have the fine KS2 scores from FFT for this cohort of students and side by side they match in 200 out of 213 students, those that do not are negligible. I decided to use the FFT KS2 Fine Levels and loaded into SISRA and unsuprisingly ended up with the same value of -0.34

 

I can now only suggest that the Attainment which FFT is using is wrong, in fact i have just re read the blurb for the Tables Checking/Progress 8 data and it is using Amended data.

 

When will RAISE be using the amended data

 

Mark

[CAM]

I'm getting weird results from FFT but not sure if it's the same problem when exporting data from Aspire.

 

There are 4 columns with KS2 Fine Scores: Reading, Writing, Maths and Science. There is a fifth column called RWM which according to notes is an average of all the columns. But when I average the four fine grade columns with an =AVERAGE( ) formula the numbers don't add up to the pre-populated RWM column. Even trying different combinations like averaged English score and Maths only doesn't add up.

 

Are they doing something weird when working out the data?

[hschram]

Hi Cam,

 

I believe that for the current year 11 students KS2 fine point score is calculated this way.

Reading+Writing=English and Maths=Maths (English+Maths)/2 = APS for English and Maths

 

If you click on the top entry in the search result below you will get an excel sheet that does the calculations for you as long as you got the KS2 data.

https://www.google.co.uk/search?q=ks2+2010+marks+fine+levels&sourceid=ie7&rls=com.microsoft:en-GB:IE-Address&ie=&oe=&safe=active&gfe_rd=cr&ei=B70sVcy8DMbH8gf6nYCwDg&gws_rd=ssl

 

I hope this helps.

Helgi

[CAM]

Thanks for the help. So I ignore Science entirely?

[hschram]

Hi Cam,

 

As far as I am aware of then yes ignore KS2 science grades for your current Y11 students.

 

Helgi

[ICTLady]

Resurrecting an old thread here........... the link that @LosOjos kindly posted above does not seem to work any more. Here's an updated link to the article http://www.fft.org.uk/FFT/media/fft/News/KS2_Reliability_-_Validity_MT_FINAL.pdf

 

Does anyone know whether FFT still calculate fine levels differently to the DfE?

[mark80]

^^ This may be entirely hearsay but I have been told they round the fine points differently to DFE

[Corvidae]

I also found a difference in the FFT KS2 data and have now switched over to using the KS2 data from KtS exclusively and calculating it to the DfE guidance.

 

So far I'm finding the offerings from the various data analysis systems to be inadequate in how they display the P8 data so I've also set up a Progress 8 worksheet to check that their headlines are accurate and to make the subjects included in the P8 calculation for each student visible by year/band/class so staff can better target their interventions.

[CAM]

It's not inadequate, it's a total mess. We have multiple sources of data with different spins and twists that throw our predictions off. Right now I have KS2 data in 4Matrix that is TA and Test Levels from KtS and fine data from FFT as a decimal (which I'd never even heard of as I'd been using SubLevels in the test field). So....I'm lost and I've been working with data for years. I feel sorry for anyone new trying to get these rather critical results in.

 

I am not confident at all that what I have in my models is accurate anymore. :/

[bobsmith]

fine levels is the way to go, 4matrix is much happier using them and it's all that SISRA will accept.

 

There are a load of articles on the net warning against forecasting Progress 8, however using Progress 8 methodology can still generate useful lists of students who are "on average" under-performing.

It's just the final figures produced that are dubious.

[CAM]

Fine levels is decimal? This is where I get confused....

[bobsmith]

ok, in a nutshell this is what happens:

 

kid takes test - gets marks out of something or other (let's pretend it's 100).

 

using grade boundaries the student is given a whole level.

 

when it comes to secondary we then look at the actual marks again, and we work out how far through the level the marks come out, so we get a fine level like 4.56

 

If multiply it by 6 we get a fine points score e.g. 27.36

 

sub level is just another lookup from the fine level.

[ICTLady]

The advice on how to calculate the fine level is included in the Progress 8 technical documentation, p.16: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/497937/Progress-8-school-performance-measure.pdf

 

I still can't understand why the DfE have made this information so difficult for DMs to get their hands on.

[mark80]

^^ Agree 100% with that!

 

My gripe with the DFE comes when trying to marry up the GCSE APS they use as a starting point for the L3VA (6th form) and the data checking results csv file for the GCSE results from the summer. You would think APS is straightforward but i have yet to get a match!

[CAM]

Sounds like don't worry if it's wrong, it won't match anyway. I have a feeling the DfE aren't fans of us giving predictions. :/

[NormalDistribution]

Schools are at the thin end of the data wedge.