Work ADR Reader
Four systems read the same medical notes looking for drug side effects. The most advanced one finished last.
Python · PubMedBERT · LLM eval
0.83
Specialist medical model
0.79
Plain keyword matcher
0.73
General-purpose AI model
0.71
General LLM, no training on the task
The finding
A specialist medical model won (F1 0.83). A plain keyword matcher (0.79) beat a general-purpose AI model (0.73), and a general LLM given no training on the task came last (0.71) — while inventing details that were not in the text.
Method
One held-out test set across TF-IDF, MiniLM embeddings, PubMedBERT and a zero-shot LLM; hallucination rate measured rather than assumed.
So what
Buying the newest model is not a strategy. Fit to the domain and the task beat sophistication here, and the only way to know that in advance is to measure it on your own data before committing to it.
Interactive
ADR READER · LEADERBOARD
Four models, one test set.
F1 score on the 'Related' (ADE) class — how well each model spots a real adverse drug event without crying wolf on a harmless sentence.
The domain-tuned model wins — but a plain keyword matcher beats a general-purpose transformer, and the zero-shot LLM comes last despite being the most sophisticated model on paper. See the examples below for why.
ADR READER · EXAMPLE EXPLORER
Pick a sentence. Watch four models disagree.
Real cases from my test set — the drug and side effect are highlighted, and each model's verdict is shown in plain English, not a raw 1 or 0.
RESULTS: Evidence of neurological improvement and rehabilitation potential after severe myelopathy due to intrathecal injection of doxorubicin.
What the LLM actually extracted
Drug: doxorubicin · Effect: neurological improvement
The hard trap: the 'improvement' framing fooled the domain model AND the LLM — but the simple keyword model caught the harm. Even the best model slips on tricky sentences.
Hepatotoxicity was observed in three patients treated with isoniazid.
A clear-cut adverse drug event — all four models agree.
Lithium was effective in reducing manic episodes, with no adverse effects reported.
A clear non-event (the drug helped) — all four models agree.
On cessation of the injections, the retrocorneal membrane grew rapidly to involve the entire posterior cornea.
What the LLM actually extracted
Drug: — · Effect: retrocorneal membrane growth
The LLM cried wolf — it flagged a harm with no named drug, contradicting itself and the label.
Full write-up
What I did
Adverse drug event (ADE) detection in free-text medical notes is a common pharmacovigilance task, so I ran the same test set through four models of increasing sophistication and watched where each one actually broke.
Lap 1 — keyword matching. A plain TF-IDF model, just matching vocabulary. No understanding of context, but it’s a strong, cheap baseline that a lot of production systems still lean on.
Lap 2 — general AI. A general-purpose sentence embedding model (MiniLM), pretrained on ordinary web text. It understands meaning better than keyword matching, but has never specifically learned the vocabulary or framing of clinical writing.
Lap 3 — domain AI. A biomedical transformer (PubMedBERT), pretrained on millions of medical abstracts. It knows this exact register of language — how clinicians phrase harm, and how they phrase incidental findings.
Lap 4 — the LLM. A zero-shot general-purpose LLM, given no fine-tuning on this task at all, just a prompt asking it to judge each sentence and extract the drug and effect it found.
Key finding
The domain-tuned model won overall, but the ranking beneath that is the interesting part: a simple keyword model beat the general-purpose transformer, and the zero-shot LLM came last — despite being the most sophisticated model on paper. See the leaderboard and worked examples below for exactly where each one broke.
The zero-shot caveat
The LLM here was given zero task-specific tuning — no fine-tuning, no few-shot examples, just a prompt. That’s the fairest comparison of “out of the box” capability, but it’s also the LLM’s worst-case setup: a few-shot or fine-tuned version would likely close some of this gap. The honest reading isn’t “LLMs are bad at this” — it’s that a zero-shot LLM is not a drop-in replacement for a model actually trained on the task, and deploying one without checking its failure modes first is a real risk in a domain where a missed or hallucinated harm has consequences.