Calculated grades, predicted grades, forecasted grades and actual A-level grades: Reliability, correlations and predictive validity in medical school applicants, undergraduates, and postgraduates in a time of COVID-19

Abstract

Calculated A-level grades will replace actual, attained A-levels and other Key Stage 5 qualifications in 2020 in the UK as a result of the COVID-19 pandemic. This paper assesses the likely consequences for medical schools in particular, beginning with an overview of the research literature on predicted grades, concluding that calculated grades are likely to correlate strongly with the predicted grades that schools currently provide on UCAS applications. A notable absence from the literature is evidence on whether predicted grades are better or worse than actual grades in predicting university outcomes. This paper provides such evidence on the reduced predictive validity of predicted A-level grades in comparison with actual A-level grades.

The present study analyses the extensive data on predicted and actual grades which are available in UKMED (United Kingdom Medical Education Database), a large-scale administrative dataset containing longitudinal data from medical school application, through undergraduate and then postgraduate training. In particular, predicted A-level grades as well as actual A-level grades are available, along with undergraduate outcomes and postgraduate outcomes which can be used to assess predictive validity of measures collected at selection. This study looks at two UKMED datasets. In the first dataset we compare actual and predicted A-level grades in 237,030 A-levels taken by medical school applicants between 2010 and 2018. 48.8% of predicted grades were accurate, grades were over-predicted in 44.7% of cases and under-predicted in 6.5% of cases. Some A-level subjects, General Studies in particular, showed a higher degree of over-estimation. Similar over-prediction was found for Extended Project Qualifications, and for SQA Advanced Highers.

The second dataset considered 22,150 18-year old applicants to medical school in 2010 to 2014, who had both predicted and actual A-level grades. 12,600 students entered medical school and had final year outcomes available. In addition there were postgraduate outcomes for 1,340 doctors. Undergraduate outcomes are predicted significantly better by actual, attained A-level grades than by predicted A-level grades, as is also the case for postgraduate outcomes.

Modelling the effect of selecting only on calculated grades suggests that because of the lesser predictive ability of predicted grades, medical school cohorts for the 2020 entry year are likely to under-attain, with 13% more gaining the equivalent of the current lowest decile of performance, and 16% fewer gaining the equivalent of the current top decile, effects which are then likely to follow through into postgraduate training. The problems of predicted/calculated grades can to some extent, although not entirely, be ameliorated, by taking U(K)CAT, BMAT, and perhaps other measures into account to supplement calculated grades. Medical schools will probably also need to consider whether additional teaching is needed for entrants who are struggling, or might have missed out on important aspects of A-level teaching, with extra support being needed, so that standards are maintained.

“… the … exam hall [is] a level playing field for all abilities, races and genders to get the grades they truly worked hard for and in true anonymity (as the examiners marking don’t know you). [… Now we] are being given grades based on mere predictions.” Yasmin Hussein, letter to The Guardian, March 29^th 2020 [1].

“[Let’s] be honest, this year group will always be different.” Dave Thomson, blogpost on FFT Educational Lab [2]

“One headmistress commented that ‘entrance to university on teachers’ estimates may be fraught with unimagined difficulties’. … If there is in the future considerable emphasis on school assessment, some work of calibration is imperatively called for.” James Petch, December 1964[3].