GCSE Geography AI marking, 1 to 9 marks

Across 104 GCSE Geography exemplars, DeepMark matched the definitive mark exactly on 71 and was within 1 mark on 94. The average gap was 0.4 marks. Every answer received the same mark on all 2 runs.

Short questions are where DeepMark is closest. On questions worth 1 to 4 marks it matched the definitive mark exactly on 53 of 60 exemplars and was within 1 mark on 59.

1. 1We collected real GCSE Geography answers that the exam board publishes with the mark it awarded, across all three papers of one exam series.
2. 2We loaded each question paper, its resource booklet and its mark scheme into DeepMark from the original documents.
3. 3Answers were typed up from the published documents, including answers that rely on a figure or map from the paper.
4. 4Each answer was marked 2 times by the same marking engine teachers use. Nothing was tuned on these answers.
5. 5We compared DeepMark's whole mark with the definitive mark for every answer. On 9 mark questions we compared the content mark; spelling and grammar marks were left out.

The study covers point marked questions, where each correct point earns a mark, and level marked questions worth 4, 6 and 9 marks, where the answer is placed in a level first. DeepMark was within 1 mark on every 2 mark and 4 mark question in the study.

On the longer, level marked questions, each dot is a pair of marks: the definitive mark along the bottom, DeepMark’s mark up the side. Dots on the dashed line are exact matches.

Every exemplar in the study, with the definitive mark and DeepMark’s mark. We publish marks only, never student writing.

Average gap

How far DeepMark's mark was from the definitive mark, on average, ignoring direction. A gap of 1.3 marks means a typical answer was a little over 1 mark away.

Matched exactly

DeepMark gave the same whole mark as the definitive mark. This is the strictest test, and only possible because every DeepMark mark is a whole mark.

Within 1 mark

DeepMark's mark was the definitive mark or 1 mark either side.

Ranking correlation

How closely DeepMark puts answers in the same order as the definitive mark, from 0 to 1. It measures order, not accuracy, so we always show it beside the average gap, never instead of it.

• AI marking repeatability264 answers, 76 papers, 185 questions, 48 teachers, 1,320 runs
• GCSE History AI marking, up to 16 marks57 exemplars on questions worth 4, 8 and 16 marks
• GCSE English Language reading AI marking, 15 marks31 exemplars on 15 mark reading questions
• GCSE Business AI marking, whole mock papers23 handwritten mock scripts on a 43 mark paper
• A level History AI marking, 25 marks44 exemplars