Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models

Abstract

Introduction. Large language models (LLMs) are being explored for multiple applications in medical research, including medical text classification. Authors of the study evaluated the performance of three off-the-shelf LLMs in classifying unstructured free-text CT pulmonary angiography reports for pulmonary embolism (PE)-related diagnostic labels.

Methods. Authors assessed human-labeled CT reports using three LLMs (ChatGPT-4o, Llama 3.3 70B, and Llama 3.1 8B) with zero-shot simple and chain-of-thought (CoT) prompts. Classification tasks included acute PE (N = 1,025) and, among reports with PE (n = 660), bilateral PE and large PE.

Results. LLMs demonstrated high classification performance for acute PE and bilateral PE but lower performance for large PE. With simple prompts, the highest F1 scores were 97.0% for acute PE (Llama 3.1 8B; sensitivity, 95.6%; specificity, 97.3%), 92.9% for bilateral PE (ChatGPT-4o; sensitivity, 98.0%; specificity, 88.6%), and 72.8% for large PE (ChatGPT-4o; sensitivity, 82.9%; specificity, 82.2%). CoT prompting did not meaningfully improve acute PE classification but provided slight improvements in subclassification performance for bilateral and large PE.

Conclusions. Off-the-shelf LLMs can classify free-text CT reports using simple prompt engineering and may facilitate automated data extraction for medical research following appropriate evaluation of classification performance.