Problem
A screening model reporting QWK 0.90 says nothing about whether it read the retina or the camera that took the picture, whether it holds at another clinic, or whether its errors belong to the model or to the labels it learned from.
selected case study
Diabetic Retinopathy Grading (APTOS-2019)
A retinopathy grading study on 3,662 fundus photographs that tests every claim it makes: whether the model reads the retina or the camera, whether it holds on two other populations, and - when it failed on one - whether the cause was the model or its labels.
Quick technical review
The case study in four points.
Architecture
An installable package with a stage pipeline that refuses to run out of order, a shared preprocessing module with a provenance manifest per image cache, EfficientNet-B0 ordinal regression under resumable five-fold cross-validation, and MLflow…
Read complete note
An installable package with a stage pipeline that refuses to run out of order, a shared preprocessing module with a provenance manifest per image cache, EfficientNet-B0 ordinal regression under resumable five-fold cross-validation, and MLflow tracking exported to a versioned runs file. The ensemble is exported to ONNX and served by FastAPI on Render with weights on Hugging Face.
Results
3,662 fundus images; five-fold ensemble QWK 0.9091
Limitations & scope
The finer hypothesis failed: the whole Moderate grade moved, not the subtle cases specifically, so the model redrew a boundary rather than learning new features. The fine-tuned model over-grades APTOS, so the deployed model is unchanged, and the…
Read complete note
The finer hypothesis failed: the whole Moderate grade moved, not the subtle cases specifically, so the model redrew a boundary rather than learning new features. The fine-tuned model over-grades APTOS, so the deployed model is unchanged, and the service states that its referral flag under-calls Moderate disease without exudates. It is not a medical device.
Summaries reuse the project’s documented evidence. Review the complete case study for evaluation boundaries.
Evidence-led visual review
Explore the actual work.
Actual project screenshots and repository figures are shown with source provenance.
Dated demo screenshot
Public demonstration screenshot dated 2026-10-12; does not establish that the demo is currently online.
Open published source3,662 fundus images; five-fold ensemble QWK 0.9091
External referable AUC: IDRiD 0.984 / Messidor-2 0.819
Adjudicated-label fine-tuning: AUC 0.830 → 0.925; control 0.834
Metadata-only shortcut reaches QWK 0.652 without reading the retina
Project proof
Verified external evidence
Last verified 2026-10-05
Problem and approach
Problem
A screening model reporting QWK 0.90 says nothing about whether it read the retina or the camera that took the picture, whether it holds at another clinic, or whether its errors belong to the model or to the labels it learned from.
Architecture
An installable package with a stage pipeline that refuses to run out of order, a shared preprocessing module with a provenance manifest per image cache, EfficientNet-B0 ordinal regression under resumable five-fold cross-validation, and MLflow tracking exported to a versioned runs file. The ensemble is exported to ONNX and served by FastAPI on Render with weights on Hugging Face.
Validation
Five-fold cross-validation over the leak-cleaned pool with the test split held out: 0.8902 plus or minus 0.0086 per fold, 0.9091 for the five-fold ensemble. Preprocessing ideas were compared on paired folds and both came back null - CLAHE p 0.865, squash p 0.468 - as did folds stratified on image resolution.
Confound check
A classifier trained only on file metadata reaches QWK 0.652 without seeing a retinal pixel, because 92.5% of the images at the most common resolution are graded healthy. Duplicate images disagree 29.1% of the time, putting a single label near 84% accuracy.
External validation
Two pre-registered tests. On IDRiD, where the shortcut is structurally unavailable, the prediction that a shortcut-dependent model would over-call disease failed in the model’s favour: referable AUC 0.984, specificity 0.987. On Messidor-2, graded by a panel of three retina specialists, the prediction of AUC 0.93 or better failed: 0.819, with 83% of Moderate eyes graded below the referral line.
The cause, tested
A third pre-registered experiment fine-tuned the five fold models on half of Messidor-2, split by patient. Adjudicated labels lifted the held-out half from AUC 0.830 to 0.925; a control arm trained on the same images with the model’s own grades reached only 0.834. The failure was the training labels, not the camera.
What is not claimed
The finer hypothesis failed: the whole Moderate grade moved, not the subtle cases specifically, so the model redrew a boundary rather than learning new features. The fine-tuned model over-grades APTOS, so the deployed model is unchanged, and the service states that its referral flag under-calls Moderate disease without exudates. It is not a medical device.
Delivery
177 tests including an end-to-end training run on synthetic images, a 65% coverage floor in CI across three Python versions, and an ONNX service at a 204 MB peak on a 512 MB host whose grades match the offline ensemble on 9 of 10 held-out images.
Why it matters
It treats a headline metric as a claim to test and writes predictions down before the answer, so the results that went against the model carry the same weight as the ones that went for it.
Architecture and analysis
- Measures a metadata-only shortcut baseline and a label ceiling before trusting any score.
- Commits three predictions before the data is touched and reports each one whichever way it came out.
- Isolates a transfer failure to label quality with a same-image control arm.
Testing and validation
- Installable package with a precondition-checked stage pipeline and MLflow tracking
- Resumable cross-validation that rebuilds the ensemble from persisted fold predictions
- ONNX export with parity checks, served by FastAPI with every page number injected from the model card
- 177 tests with a 65% coverage floor, CI on three Python versions
Operational value
- Five-fold cross-validation on the 3,247-image leak-cleaned pool of the 3,662-image dataset, with the 366-image test split held out: 0.8902 plus or minus 0.0086 per fold, ensemble QWK 0.9091.
- Shows a metadata-only model reaching QWK 0.652 without reading the retina, and a label ceiling near 84%.
- IDRiD referable AUC 0.984 and specificity 0.987 on a population where the shortcut is unavailable.
- Messidor-2 referable AUC 0.819 against adjudicated labels, a pre-registered prediction that failed.
- Fine-tuning on adjudicated labels: held-out AUC 0.830 to 0.925, while the same-image control reaches 0.834.
How to inspect
- Read the write-up first; it tells the three pre-registered experiments in order.
- Open the live demo and read its limitations section before uploading an image.
- Compare the pre-registration commits in docs/ with the outcome sections below them - the predictions were pushed before the data existed locally.