AI OBSERVATORY / SIGNALS

AIO-S0006 · 25 OCT 2026

A randomised trial at one eye hospital: an LLM agent takes better histories than residents

We observe a randomised trial in which an LLM agent outperformed residents on history-taking, with no difference on test recommendations.

AIO Research Team · The AI Observatory

+17.7
Points higher on history-taking quality (0 to 100 scale), agent against residents [1]
172
Patients randomised [1]
n.s.
Difference in test recommendations [1]

Coverage: one hospital, China (People's Republic of) · Source: npj Digital Medicine, 5 Oct 2026

What we observe

In a randomised clinical trial at a tertiary ophthalmic hospital in Guangzhou, China, 172 of the 175 patients approached were randomised 1:1 to history-taking before consultation by either an LLM agent or ophthalmology residents, between 10 May and 10 June 2025 [1]. The agent scored higher on pre-consultation quality (MedHistory score: adjusted mean difference 17.7 points on a 0 to 100 scale, 95% confidence interval 14.1 to 21.3) [1]. Performance on test recommendations did not differ significantly across stages [1].

In an exploratory analysis of history alone, the agent's diagnostic F1 score was 0.85 against 0.68 for residents; when ocular signs were added, residents improved more, narrowing the gap [1]. Median interaction time was 11.2 minutes for the agent against 3.1 minutes for residents [1].

The agent gathered better histories; residents narrowed the gap once ocular signs were added.
Diagnostic F1 score from history alone (exploratory analysis) [1]Bar chart. LLM agent 0.85; residents 0.68. F1 scale 0 to 1. Randomised trial, 172 patients, one eye hospital. Source: Luo et al., npj Digital Medicine, 5 Oct 2026.Diagnostic F1 score from history alone (exploratory analysis) [1]LLM agent0.85Residents0.6801F1 score, 0 to 1. Source: Luo et al., npj Digital Medicine, 5 Oct 2026 [1].

Why it matters

The benefit is specific: standardised data gathering before a consultation, not diagnosis or treatment. The authors conclude that the findings support a hybrid workflow in which clinicians focus on physical examination and diagnostic synthesis [1]. The trial ran at one hospital, in one specialty, with non-emergency patients, so it does not generalise on its own. Registration: ClinicalTrials.gov NCT06824389 [1].

Method

Source: the peer-reviewed article in npj Digital Medicine, published online 5 Oct 2026, read on 25 Oct 2026 [1]. Figures are from the abstract as published. The chart shows the exploratory F1 result as reported. Framing follows AIO Watch method note AIO-M01, v1.2 [2].

References

  1. [1] Luo, M.J., Lai, Y., Wang, W. et al. "Standardized pre-consultation by a large language model agent vs ophthalmology residents: a randomized clinical trial". npj Digital Medicine, 5 Oct 2026. doi:10.1038/s41746-026-03232-x. https://www.nature.com/articles/s41746-026-03232-x. Accessed 25 Oct 2026.
  2. [2] AIO Research Team. AIO Watch method note AIO-M01, v1.2, 25 Oct 2026.
AIO Research Team · AIO-S0006 · Primary sources accessed 25 Oct 2026

Open Signal (PDF) All Signals