How Running Paper marks an assessment
Running Paper is designed to mark handwritten, pen-and-paper assessments at scale.
Print the assessment from the platform and have students complete it on paper. Once they have finished you can bulk scan all the paper scripts into a single file in any order. Upload that file to Running Paper and it automatically identifies the assessment, sorts the pages, matches each page to the correct student and reconstructs each student’s complete script. Once allocated, every assessment is then marked independently by two different AI models: one as the primary marker and one as the validator.
The primary model produces the working mark. The validating model independently marks the same question. We then compare the two marks question by question. Where the models disagree beyond the tolerance set for that question, the question is flagged as low confidence. The teacher can then review the response, the relevant part of the mark scheme and the mark that has been proposed.
Accuracy has to be measured against something
Mark schemes require interpretation, and two experienced examiners can legitimately disagree about how an answer should be awarded. To address this, we create a set of benchmark scripts: exam scripts marked to an agreed standard that serves as our ground truth.
Benchmark scripts are independently marked by former exam board examiners and experienced subject leads. Where those markers disagree, the question is moderated and an agreed position is established. That gives us a reference point against which we can measure a marking configuration.
It also means that we are not asking whether an AI model agrees with another AI model and calling that accuracy. We are asking how closely the system performs against independently established expert marking.
How we benchmark a marking configuration
A marking configuration is more than just the name of an AI model. It specifies which models are used and in what role, which one acts as the primary marker and which as the validator, along with decoding settings such as temperature and reasoning effort (where the model supports it), and the construction of the prompt. It also covers whether the pipeline uses retrieval-augmented generation (RAG) and whether the models are fine-tuned on our own data.
When we have a new model, a new model version or a new configuration, we take our held-out benchmark scripts and run the new configurations against them. The scripts are marked blind. The new configuration does not know the definitive marks. We then compare the results against the benchmark. This lets us answer the following questions:
Does this new configuration perform better than, or at least as well as, the one we are already using? How does it compare against expert examiner marking?
Testing more than just the model
We can test different models and different configurations. That can include changing the model itself, changing temperature, changing reasoning effort, changing the way retrieval-augmented models are configured, or using different versions of our own retrieval-augmented or fine-tuned approaches.
The reason for testing these settings systematically is simple: we do not assume that a newer model is automatically a better marking model. A new model may be stronger in general reasoning but worse at a particular subject or question type. A change in configuration may reduce errors in one area while increasing them in another. The benchmark gives us a way to measure the effect.
Simulating the marking process
One of the most important parts of the Running Paper approach, and a core part of the accuracy and benchmarking framework we have built, is the use of simulations.
A simulation takes the results from different model configurations and asks what would have happened if they had been used together as a primary marker and validator. We choose a primary and a validating model configuration, run each independently against the benchmark, then compare their results question by question.
For each simulation, we can calculate measures including:
- Mean absolute error: how far the primary marking is from the definitive benchmark marks on average.
- Percentage within tolerance: how often the total mark for a student’s whole script falls within an acceptable percentage tolerance of the definitive benchmark total.
- Percentage of questions flagged: how often the primary and validating models disagree enough to trigger a low-confidence review.
- Percentage of errors caught: how many of the primary model’s genuine errors would have been identified by the validator.
- Residual error: how many errors remain after the validation mechanism has done its job.
Why the two-model approach matters
Every question is marked independently by two models. The validator does not see the primary model’s mark, reasoning or confidence. That independence is deliberate. If the second model simply saw what the first model had already decided, we would learn much less from the comparison. The system is therefore designed to catch disagreement.
It is important, however, to be clear about the limitation. Two models can agree and still be wrong. This is known as correlated error. That is why Running Paper does not treat agreement between two models as proof that a mark is correct. The benchmark process is what allows us to test how often the system is actually right, including the errors that survive the validation process.
A real example: GCSE Physics Higher
To make this concrete, here is an example of one of our benchmark assessments and simulations:
Configuration
| Exam paper | GCSE Physics Higher |
| Paper total marks | 63 marks |
| Exam scripts | 100 |
| Primary model | rp-gemini-rag |
| Validating model | rp-gpt-rag |
Both models in this configuration are Running Paper retrieval-augmented models, built on different underlying frontier models from different providers.
Results
| Measure | Result |
|---|---|
| Questions flagged as low confidence | 19.1% |
| Mean absolute error (MAE), primary model, before review | 1.1 marks |
| Mean absolute error (MAE), after low-confidence review | 0.5 marks |
| Marks within one-mark tolerance, after review | 95% |
| Scripts within two-mark tolerance, after review | 100% |
Reading these figures
The flag rate of 19.1% means roughly one question in five was flagged as low confidence.
The reduction in mean absolute error from 1.1 to 0.5 marks is the low-confidence mechanism working as designed. The MAE of 1.1 marks is the primary model marking alone. The MAE of 0.5 marks is after the low-confidence questions have been corrected.
It is worth stating plainly: the accuracy that matters is not the model’s accuracy. It is the accuracy of the model, the validator and the teacher working together. The residual 0.5 marks is not zero and we do not present it as such. It represents errors that survived both the validation comparison and teacher review.
The post-review MAE (0.5), the 95%, and the 100% figures above are drawn from a simulation that assumes every flagged question is corrected to the definitive mark. They show the accuracy the system is designed to achieve once low-confidence questions are addressed.
What happens when we introduce a new model?
Every time we introduce a new model or change an important configuration, we repeat the evaluation process. The new configuration is tested against the benchmark and compared with the current production configuration.
The benchmarking does not just tell us which single configuration is best overall. It tells us which configuration is best for a given subject and a given level. Assessments in different subjects and levels are routed to the configuration that performs best.
We are looking for continuous improvement. This means that our production models are not selected once and then left alone. They are continually challenged against new model configuration sets. Where a new configuration meets the required threshold, and where it provides an improvement, it can then be promoted into production as shown by the cycle below.
Our benchmark and simulation database
We have benchmark assessments and simulations from KS3 to KS5 and across a wide variety of subjects such as maths, English, science, humanities, religious studies, MFL and more.
If you would like to know how the system performs on your specific subject, get in touch via the contact form or book a demo below.
