Artificial intelligence in thoracic surgery consultations: evaluating the concordance between a large language model and expert clinical decisions

Abstract

Background: Artificial intelligence (AI) and large language models (LLMs) are increasingly used in clinical workflows, but their real-world application in thoracic surgery decision-making remains underexplored.

Methods: This retrospective observational study assessed the concordance between diagnostic and therapeutic recommendations generated by Scholar GPT (based on GPT-4) and decisions made by board-certified thoracic surgeons. All outpatient consultations over one week in a tertiary care hospital were included. Each case was evaluated using a 6-point concordance scale (0–5), developed to quantify agreement in diagnosis and treatment planning. This was a retrospective observational, single-centre analysis; two independent thoracic surgeons assigned the concordance score. We report descriptive statistics and used t-tests/ANOVA for continuous variables and chi-square tests for categorical variables. Given the exploratory design, no a priori sample-size calculation or power analysis was performed.

Results: A total of 81 consultations were analysed. The mean concordance score was 3.67 ± 1.17. High concordance (scores 4–5) occurred in 56.8% of cases, particularly in oncological diagnoses such as mediastinal and pleural tumours. Lower concordance was observed in complex or functional conditions like metastatic lung disease and thoracic outlet syndrome. No significant differences were found between consultation modalities or visit types.

Conclusion: Scholar GPT demonstrated promising alignment with surgeon decisions in structured oncologic cases but showed variability in complex scenarios. While AI may assist in streamlining outpatient workflows, its use should remain complementary to expert clinical judgment. These findings are exploratory and should be interpreted with caution given the small sample size and single-centre, one-week design.