It's all about the mark scheme

There's a question we get asked a lot: how accurate is AI marking?

It's a fair question. But it's also, we'd argue, the wrong one, or at least an incomplete one. Because the accuracy of any mark, human or AI, is only ever as good as the mark scheme used to produce it. That's what this post is about.

Marking is way harder than it looks

Most people assume marking is a relatively mechanical process. A student writes an answer. An examiner reads it. The examiner decides how many marks it's worth. Tick. Tick. Cross. Job done.

In reality, that middle step, where the examiner decides how many marks it's worth, is doing a tremendous amount of work. For essay-style questions, it requires the examiner to hold a mental model of quality, match a student's response against that model, and make a holistic judgement call. Two trained examiners working from the same mark scheme can legitimately land on different marks for the same script. Ofqual's own research shows that for GCSE English, the probability of two examiners agreeing on the exact grade is around 50 to 60%. Not because either is wrong. Because the task is genuinely interpretive, and a single question can carry two legitimate scores that reflect a difference in professional judgement.

This isn't a criticism of examiners. It's a structural feature of the way essay-style questions are assessed. And understanding why requires understanding how mark schemes are actually built.

What is an Assessment Objective (AO)?

Assessment Objectives, or AOs, are the skills that a qualification is designed to test. They're set by Ofqual, not by individual exam boards, which means they're consistent across AQA, OCR, Pearson and every other board offering the same subject.

But what each AO means varies significantly by subject. In GCSE History:

• AO1 is knowledge and understanding of the period studied.
• AO2 is explanation and analysis using second-order historical concepts like cause, consequence and significance.
• AO3 is source analysis.
• AO4 is evaluation of historical interpretations.

In GCSE Business, there are only three AOs:

• AO1, knowledge (35% of marks).
• AO2, application to a specific business context (35%).
• AO3, analysis and evaluation (30%).

In GCSE English Literature:

• AO1 covers reading and personal response.
• AO2 covers language, form and structure.
• AO3 covers context.
• AO4 covers SPaG.

The AOs answer one question: what skill are we measuring? They don't say anything about what a good or bad answer looks like. That's the job of the level descriptors.

One thing worth clearing up: the numbering is misleading. AO1, AO2, AO3 don't represent a ladder of difficulty or a sequence to follow. They're independent skill categories, administrative shorthand, essentially. The real ladder is the level descriptors within each AO, which describe how well a student executed that particular skill. In practice, particularly in holistic level of response marking, the AOs are often blended within a single response rather than addressed sequentially, which is part of what makes that marking harder to standardise.

From AOs to level descriptors: translating skills into quality

A level descriptor takes an abstract AO and describes what it looks like in practice at different qualities of response.

Take AO2 in History, explanation using historical concepts. A Level 1 descriptor might describe a response that "offers a simple explanation, with limited development." A Level 3 descriptor for the same AO might describe "a developed explanation that demonstrates a clear understanding of cause and consequence, with well-chosen supporting evidence." Here we're measuring the same skill (the AO) but with completely different qualities.

The descriptor is doing the translation work between the abstract (the AO) and the concrete (this student's answer). It gives the examiner a benchmark to hold in their head as they read.

The quality of that translation matters enormously. A descriptor that uses vague qualitative language (sophisticated, perceptive, some understanding) leaves a lot of interpretive room. A descriptor that specifies what a Level 3 response does rather than what it is gives the examiner something concrete to check against: "identifies at least two relevant causes and explains the consequence of each with reference to the period studied."

Level of response bands: where the mark actually comes from

Level of response (LoR) mark schemes organise the level descriptors into bands, each carrying a mark range. A 9-mark question might have three bands: Level 1 (1 to 3 marks), Level 2 (4 to 6 marks), Level 3 (7 to 9 marks). Each band has a descriptor. The examiner's job is to decide which band the response best fits, then award a specific mark within that band based on how securely it meets the descriptor.

This is a two-step process:

1. Apply the level.
2. Mark within the level.

The level decision is the bigger judgement call. Once that's made, the within-band mark is usually more straightforward: a response that just clears the Level 3 threshold gets 7; one that fully exemplifies it gets 9.

What makes LoR marking distinctive and harder to standardise is that the AOs are often assessed together within a single band, not separately. In English Literature, AO1 (personal response) and AO2 (language analysis) aren't scored independently. The level descriptor blends them into a single description of what a Level 3 answer looks and feels like as a whole. The examiner isn't running two parallel checklists. They're forming a single holistic impression, then matching it to a descriptor.

That's where legitimate disagreement enters the system. The descriptor uses words like "developed" and "perceptive." What clears that bar is, to some degree, a matter of professional judgement.

The three types of question, and why they're not all equal

Not all questions use LoR mark schemes. It's worth being clear about the different types, because they behave very differently under any marking system, human or AI.

Deterministic, or selected response, questions have one right answer. Multiple choice is the clearest example. The challenge here was never deciding the right answer. It was extracting the student's chosen answer from a handwritten worksheet at scale. That problem is now solved. AI can read and interpret handwritten responses in a way that simply wasn't possible before.

Point-based questions award one mark per valid point, often with more indicative points available than marks available. A 4-mark question might have six acceptable answers in the mark scheme; any four will do. The marking task here is pattern-matching: does this student's response contain enough of the expected content? These questions are relatively straightforward to mark consistently, because the mark scheme is essentially a checklist.

Level of response questions are everything we've described above. They're the hardest to mark, the hardest to standardise, and the most common in the essay-heavy humanities subjects where marking workload is most acute. They're also where the mark scheme matters most.

Why vague descriptors produce variable marks, every time, for everyone

Here's the key point, and it applies equally to human and AI marking.

If a level descriptor says "demonstrates sophisticated analysis," an examiner has to infer what "sophisticated" means in the context of this subject, this question, this mark total. Two experienced examiners with slightly different mental models of "sophisticated" will sometimes land in different bands. Both are following the mark scheme. The mark scheme just isn't precise enough to produce a single answer.

This is not a theoretical problem. It's what drives the remark statistics. In summer 2024, over 60,000 GCSE grades were changed following a review of marking. English Language and English Literature consistently account for a disproportionate share of those changes. Not because English examiners are worse, but because English LoR descriptors are among the most interpretive in the system.

An AI marking system inherits this variability directly. Feed it a vague descriptor and it will produce variable marks, for the same reason a human does: the specification is ambiguous. Feed it a precise descriptor and it produces consistent marks, for the same reason a human does: the specification is clear.

The mark scheme is the variable. Everything else follows from it.

What this means for AI marking

When teachers ask whether AI marking is accurate, they're usually asking the wrong question, or rather, they're conflating two different things.

Accuracy asks: does the AI agree with what an expert examiner would give? That's hard to measure, because the expert examiner is also working from the same ambiguous mark scheme and producing a mark that falls within a tolerance window, not a single definitive answer.

Consistency asks: does the AI give the same mark to the same response every time? That's measurable. And it's arguably more useful, because a consistent, transparent mark that you can interrogate is more valuable than a variable one that you can't.

Our own repeatability data shows a variance of ±0.16 marks across repeated runs at optimal settings. On a 9-mark question, where the human examiner tolerance window is roughly ±1 to 2 marks (the width of a full level band), that figure puts us well within the range the system considers acceptable for trained human examiners.

But here's what that number also tells us: when our variance increases on a particular question, it almost always tracks to a specific place in the mark scheme. A descriptor that's doing too much interpretive work. A boundary between levels that isn't clearly defined. The AI's consistency score is, in effect, a diagnostic on the mark scheme itself.

How DeepMark reads a mark scheme

DeepMark reads the mark scheme before it reads the student response. It identifies the relevant AOs for the question, locates the level descriptors, and understands the mark range for each band. When it encounters the student's response, it's doing the same two-step process a trained examiner does: level first, mark within level second.

The annotations DeepMark leaves on the script are the visible output of that process. They're not decorative. They show which parts of the response were read as evidence for a particular level, and where the response fell short of the next level up. That's the "proof of reading" that makes the mark interpretable rather than arbitrary.

Because our marking engine is model-agnostic (it can run across multiple AI models), we benchmark different models on both performance and cost for each marking task. We don't use one model for everything. Different question types and different mark scheme structures respond differently to different models, and we test for that.

Writing better mark schemes: what makes one AI-ready (and human-ready)

If the quality of marking is downstream of the quality of the mark scheme, the practical question for teachers is: what makes a mark scheme work?

A few things we've learned from marking at scale:

• Describe what responses do, not what they are. "A perceptive response" is a judgement. "A response that identifies the writer's use of contrast and explains its effect on the reader" is a specification. The second is easier to apply consistently.
• Define the boundary between levels explicitly. The hardest marking decision is always whether a response clears the threshold from Level 2 to Level 3. A mark scheme that spells out what that threshold looks like, what a response needs to do to move up rather than just what Level 3 looks like once you're there, dramatically reduces variability.
• Be specific about context in application questions. In Business, Geography, and similar subjects, AO2 marks require the student to apply knowledge to a specific scenario. A mark scheme that names what contextualised application looks like for this question, not just "applies to context" generically, gives both the examiner and the AI something concrete to check.
• Indicative content is a floor, not a ceiling. For point-based questions, the indicative content list should be read as examples, not an exhaustive list. A mark scheme that says "accept any reasonable alternative" is more useful than one that doesn't, because it signals that the marker, human or AI, has permission to reward valid responses that aren't on the list.

The honest truth about accuracy vs consistency

What we can say now is this: the question of whether AI marks accurately is harder to answer than it sounds, because human marking is not a perfect benchmark. The system itself operates within a tolerance window. It acknowledges that two trained examiners can legitimately disagree. What it requires is not perfection but consistency within an accepted range.

On that measure, consistency within the range the system considers acceptable, our data is encouraging. And the mechanism that drives that consistency is the same mechanism that drives human marking consistency: a precise, well-structured mark scheme that gives the marker something concrete to work with.

It's always been all about the mark scheme. We've just made that relationship visible.