Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
Giordano de Pinho Souza, Glaucia Melo, Josefino Cabral Melo Lima, Daniel Schneider
Why It Matters
What makes this one worth your time
This research addresses the critical gap in clinical AI evaluation for non-English languages, enhancing global accessibility and applicability of language models in healthcare.
A pioneering bilingual benchmark for clinical language models in Brazilian Portuguese.
Summary
The paper introduces ClinicalBr, a bilingual benchmark for evaluating clinical decision-making in Brazilian Portuguese and English, based on real case reports, and assesses the performance of various models across multiple clinical tasks.
Key contributions
- Development of ClinicalBr, the first bilingual benchmark for clinical decision-making.
- Evaluation of multiple language models across four distinct clinical tasks.
- Insights into task-dependent performance variations between Portuguese and English.
Notable insights
- The performance gap between Portuguese and English is task-dependent, highlighting the nuanced challenges in clinical language processing.
- Brazilian-endemic conditions are easier for models to handle than the broader corpus, suggesting effective representation in pre-training.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.07853v1 Announce Type: cross Abstract: Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted in English, and cross-lingual evaluation is needed to tackle the language gaps in global access. We introduce ClinicalBr, the first bilingual benchmark for clinical decision built from real Brazilian case reports. The corpus contains 2,892 cases drawn from 28 SciELO medical journals, spanning 18 specialties, and is structured as parallel Portuguese-English pairs. Each case supports four evaluation tasks: diagnosis retrieval, differential diagnosis, exam recommendation, and treatment planning. We evaluate four models: MedGemma-27B, Sabi\'a-4, DeepSeek-R1, and o3-mini, across both languages. The central finding is that the Portuguese-English performance gap is task-dependent, not general. In diagnosis retrieval, English yields a consistent advantage across all models, with +7.5-12.1 accuracy points. This advantage disappears in differential diagnosis, exam recommendation, and treatment planning, where confidence intervals cross zero for most models and Portuguese completeness scores are marginally higher. Brazilian-endemic conditions proved easier than the full corpus, not harder, indicating that tropical presentations are adequately represented in current pre-training. Exam recommendation was the hardest task across all models and both languages, with F1 scores below 0.10, well below the differential diagnosis ceiling of 0.20-0.27.