An AI skin cancer comparison found that an older image-recognition model outperformed two advanced chatbots at classifying skin lesions. In the reported test, GPT-6 Astra and Claude Fable 5.1 each selected the correct diagnosis in roughly 42% of cases.
By comparison, a specialised version of Google’s Inception v3 correctly classified 125 of 200 lesions, or 62.5%. The findings highlight a gap between general-purpose AI capabilities and performance on a specific medical imaging task.
Dermatologist Tests Three Models on Confirmed Lesions
Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy’s Hospital, conducted the comparison. He also holds an Honorary Reader position at King’s College London.
For the test, Lynch used 200 biopsy-confirmed lesions from the public MILK10k dataset. Half were benign, while the other half were cancers or precancerous lesions.
The systems had to choose among six diagnoses. These included melanoma, basal cell carcinoma, squamous cell carcinoma, actinic keratosis, benign moles and seborrhoeic keratoses.
Importantly, the reported accuracy measures whether each model selected the correct specific diagnosis as its first choice. It does not directly measure how reliably a system distinguishes all cancers from harmless lesions.
Older Model Leads the AI Skin Cancer Comparison
Lynch fine-tuned Inception v3 using 68,000 skin images and a single H100 GPU. According to his report, the training took approximately 20 minutes.
The architecture is around a decade old and small enough to run on a smartphone. Nevertheless, the specialised model produced substantially more correct answers than either chatbot.
Across the 200 test images:
- Inception v3: 125 correct diagnoses, or 62.5%.
- Claude Fable 5.1: 85 correct diagnoses, or 42.5%.
- GPT-6 Astra: 84 correct diagnoses, or 42%.
The test images did not form part of the CNN’s training data. However, the account does not establish whether either chatbot previously encountered material from the public dataset.
Benign Lesions Expose a Major Performance Gap
The largest differences appeared when the models assessed harmless moles and seborrhoeic keratoses.
Inception v3 correctly identified 63 of the 100 benign lesions. Meanwhile, Claude Fable 5.1 identified 39 correctly, and GPT-6 Astra identified 29.
The models also showed a tendency to favour particular answers. For example, GPT-6 Astra labelled 75 images as basal cell carcinoma, although only 25 lesions belonged to that category.
Similarly, Claude Fable 5.1 selected actinic keratosis for 53 images, despite the test containing only 20 such cases.
According to Lynch, all three systems struggled to identify melanoma as their first-choice diagnosis. Therefore, the older model’s stronger overall result does not establish reliable melanoma detection.
Chatbots Frequently Recommended Medical Assessment
Lynch also examined how the chatbots responded when asked whether a user should see a doctor.
They recommended medical assessment in nearly every case, including for benign lesions. This suggests that their referral advice did not closely distinguish harmless findings from more concerning ones in this test.
However, referral behaviour and diagnostic accuracy measure different things. Advising someone to seek assessment does not mean the chatbot correctly identified the lesion.
Small Test Highlights Limits of General-Purpose AI
Lynch acknowledged that 200 lesions cannot reliably rank models with closely matched results. Nevertheless, he argued that the sample revealed meaningful patterns in their answers.
The AI skin cancer comparison also involved a basic specialist model rather than the strongest available medical imaging systems.
According to Lynch, more advanced approaches combine image transformers and larger training datasets. He said regulated products built on these methods perform considerably better than his Inception v3 baseline.
Although he made the experimental classifier available publicly, its results do not establish readiness for clinical use. The findings concern a defined image-classification test, rather than a complete medical assessment.
0 responses to “AI Skin Cancer Test: Older Model Outperforms Chatbots”