Clinical Research

Cutting edge healthtech insights

AllClinical research7Artificial intelligence7Thought leadership1Annalise CXR5Annalise CTB3

Using AI to Identify Unremarkable Chest Radiographs for Automatic Reporting

Background: Radiology practices have a high volume of unremarkable chest radiographs and artificial intelligence (AI) could possibly improve workflow by providing an automatic report. Purpose: To estimate the proportion of unremarkable chest radiographs, where AI can correctly exclude pathology (ie, specificity) without increasing diagnostic errors. Materials and Methods: In this retrospective study, consecutive chest radiographs in unique adult patients (≥18 years of age) were obtained January 1–12, 2020, at four Danish hospitals. Exclusion criteria included insufficient radiology reports or AI output error. Two thoracic radiologists, who were blinded to AI output, labeled chest radiographs as “remarkable” or “unremarkable” based on predefined unremarkable findings (reference standard). Radiology reports were classified similarly. A commercial AI tool was adapted to output a chest radiograph “remarkableness” probability, which was used to calculate specificity at different AI sensitivities. Chest radiographs with missed findings by AI and/or the radiology report were graded by one thoracic radiologist as critical, clinically significant, or clinically insignificant. Paired proportions were compared using the McNemar test. Results: A total of 1961 patients were included (median age, 72 years [IQR, 58–81 years]; 993 female), with one chest radiograph per patient. The reference standard labeled 1231 of 1961 chest radiographs (62.8%) as remarkable and 730 of 1961 (37.2%) as unremarkable. At 99.9%, 99.0%, and 98.0% sensitivity, the AI had a specificity of 24.5% (179 of 730 radiographs [95% CI: 21, 28]), 47.1% (344 of 730 radiographs [95% CI: 43, 51]), and 52.7% (385 of 730 radiographs [95% CI: 49, 56]), respectively. With the AI fixed to have a similar sensitivity as radiology reports (87.2%), the missed findings of AI and reports had 2.2% (27 of 1231 radiographs) and 1.1% (14 of 1231 radiographs) classified as critical (P = .01), 4.1% (51 of 1231 radiographs) and 3.6% (44 of 1231 radiographs) classified as clinically significant (P = .46), and 6.5% (80 of 1231) and 8.1% (100 of 1231) classified as clinically insignificant (P = .11), respectively. At sensitivities greater than or equal to 95.4%, the AI tool exhibited less than or equal to 1.1% critical misses. Conclusion: A commercial AI tool used off-label could correctly exclude pathology in 24.5%–52.7% of all unremarkable chest radiographs at greater than or equal to 98% sensitivity. The AI had equal or lower rates of critical misses than radiology reports at sensitivities greater than or equal to 95.4%. These results should be confirmed in a prospective study. © RSNA, 2024 Supplemental material is available for this article.

2 MIN READ
Artificial intelligence

Evaluation of an Artificial Intelligence Model for Identification of Intracranial Hemorrhage Subtypes on Computed Tomography of the Head

Stroke: Vascular and Interventional Neurology, originally published 15 May 2024 Authors: James M. Hillis, Bernardo C. Bizzo, Isabella Newbury‐Chaet, Sarah F. Mercaldo, John K. Chin, Ankita Ghatak, Madeleine A. Halle, Eric L’Italien, Ashley L. MacDonald, Alex S. Schultz, Karen Buch, John Conklin, Stuart Pomerantz, Sandra Rincon, Keith J. Dreyer and William A. Mehan Background: Intracranial hemorrhage is a critical finding on computed tomography (CT) of the head. This study compared the accuracy of an artificial intelligence (AI) model (Annalise Enterprise CTB Triage Trauma) to consensus neuroradiologist interpretations in detecting 4 hemorrhage subtypes: acute subdural/epidural hematoma, acute subarachnoid hemorrhage, intra‐axial hemorrhage, and intraventricular hemorrhage. Methods: A retrospective stand‐alone performance assessment was conducted on data sets of cases of noncontrast CT of the head acquired between 2016 and 2022 at 5 hospitals in the United States for each hemorrhage subtype. The cases were obtained from patients aged ≥18 years. The positive cases were selected on the basis of the original clinical reports using natural language processing and manual confirmation. The negative cases were selected by taking the next negative case acquired from the same CT scanner after positive cases. Each case was interpreted independently by up to 3 neuroradiologists to establish consensus interpretations. Each case was then interpreted by the AI model for the presence of the relevant hemorrhage subtype. The neuroradiologists were provided with the entire CT study. The AI model separately received thin (≤1.5 mm) and thick (>1.5 and ≤5 mm) axial series as available. Results: The 4 cohorts included 571 cases of acute subdural/epidural hematoma, 310 cases of acute subarachnoid hemorrhage, 926 cases of intra‐axial hemorrhage, and 199 cases of intraventricular hemorrhage. The AI model identified acute subdural/epidural hematoma with area under the curve of 0.973 (95% CI, 0.958–0.984) on thin series and 0.942 (95% CI, 0.921–0.959) on thick series; acute subarachnoid hemorrhage with area under the curve 0.993 (95% CI, 0.984–0.998) on thin series and 0.966 (95% CI, 0.945–0.983) on thick series; intraaxial hemorrhage with area under the curve of 0.969 (95% CI, 0.956–0.980) on thin series and 0.966 (95% CI, 0.953–0.976) on thick series; and intraventricular hemorrhage with area under the curve of 0.987 (95% CI, 0.969–0.997) on thin series and 0.983 (95% CI, 0.968–0.994) on thick series. Each finding had at least 1 operating point with sensitivity and specificity >80%. Conclusion: The assessed AI model accurately identified intracranial hemorrhage subtypes in this CT data set. Its use could assist the clinical workflow, especially through enabling triage of abnormal CTs.

2 MIN READ
Artificial intelligence Annalise CTB

Breaking bias: The role of artificial intelligence in improving clinical decision-making

Cureus: Published 20 March 2023. Authors:: Brown C, Nazeer R, Gibbs A, Le Page P, Mitchell ARJ Abstract:: This case report reflects on a delayed diagnosis for a 27-year-old woman who reported chest pain and shortness of breath to the emergency department. The treating clinician reflects upon how cognitive biases influenced their diagnostic process and how multiple missed opportunities resulted in missteps. Using artificial intelligence (AI) tools for clinical decision-making, we suggest how AI could augment the clinician, and in this case, delayed diagnosis avoided. Incorporating AI tools into clinical decision-making brings potential benefits, including improved diagnostic accuracy and addressing human factors contributing to medical errors. For example, they may support a real-time interpretation of medical imaging and assist clinicians in generating a differential diagnosis in ensuring that critical diagnoses are considered. However, it is vital to be aware of the potential pitfalls associated with the use of AI, such as automation bias, input data quality issues, limited clinician training in interpreting AI methods, and the legal and ethical considerations associated with their use. The report draws attention to the utility of AI clinical decision-support tools in overcoming human cognitive biases. It also emphasizes the importance of clinicians developing skills needed to steward the adoption of AI tools in healthcare and serve as patient advocates, ensuring safe and effective use of health data. : : :

2 MIN READ
Artificial intelligence

Can an AI-driven decision aid reduce the time between chest X-rays and treatment for lung cancer patients?

Scientific presentation (RPS 1205) at ECR 2024, 28.2.-2.3.2024 in Vienna, Austria. Authors:: Cameron, Lorna NHS Grampian, Aberdeen, United Kingdom Purpose:: The primary aim of this study was to evaluate whether an AI product with the ability to identify chest x-ray (CXR) images of highest risk of lung cancer can reduce the time between imaging and treatment in those patients subsequently diagnosed with lung cancer. Method:: The NHS Grampian Innovation, Radiology and Cancer Teams collaborated with the Centre for Sustainable Delivery and the Scottish Health Technology Group to design an evaluation of the real-world impact of using an AI product designed to risk stratify CXR images. Full pathway mapping was carried out and baseline time delays between all key points (CXR, CXR reporting, CT, CT reporting, MDT diagnosis and treatment) were established. CXR images flagged as highest risk of lung cancer were expedited for CXR reporting, CT and CT reporting. NHS Grampian radiologists collaborated with the company to calibrate the product in ways that maximised identification of lung cancer whilst not overwhelming CT capacity. Results:: Several months into the project the time between CXR and CT report has dropped from 22 to 10.3 days (N=132). Radiologists identified 28 images not flagged by the product about which they were concerned about cancer. Thus far, none of these patients have been diagnosed with cancer. Under the current calibration conditions, using radiologists’ judgements, the product performs at 84.4 sensitivity and 90.5 specificity (N=24071). Conclusion:: Early results suggest AI risk stratification of CXR images may help healthcare organisations reduce the time taken to treat people diagnosed with lung cancer. This could be especially important for people who are diagnosed following CXR imaging for non-cancer reasons. In our region, this is about two thirds of people diagnosed with lung cancer.

2 MIN READ
Annalise CXR

Effects of a comprehensive brain computed tomography deep learning model on radiologist detection accuracy

European Radiology Published: 22 August 2023 Authors: Quinlan D. Buchlak, Cyril H. M. Tang, Jarrel C. Y. Seah, Andrew Johnson, Xavier Holt, Georgina M. Bottrell, Jeffrey B. Wardman, Gihan Samarasinghe, Leonardo Dos Santos Pinheiro, Hongze Xia, Hassan K. Ahmad, Hung Pham, Jason I. Chiang, Nalan Ektas, Michael R. Milne, Christopher H. Y. Chiu, Ben Hachey, Melissa K. Ryan, Benjamin P. Johnston, Nazanin Esmaili, Christine Bennett, Tony Goldschlager, Jonathan Hall, Duc Tan Vo, Lauren Oakden-Rayner, Jean-Christophe Leveque, Farrokh Farrokhi, Richard G. Abramson, Catherine M. Jones, Simon Edelstein & Peter Brotchie Abstract: Non-contrast computed tomography of the brain (NCCTB) is commonly used to detect intracranial pathology but is subject to interpretation errors. Machine learning can augment clinical decision-making and improve NCCTB scan interpretation. This retrospective detection accuracy study assessed the performance of radiologists assisted by a deep learning model and compared the standalone performance of the model with that of unassisted radiologists. Key Points: • This study demonstrated that the use of a comprehensive deep learning system assisted radiologists in the detection of a wide range of abnormalities on non-contrast brain computed tomography scans. • The deep learning model demonstrated an average area under the receiver operating characteristic curve of 0.93 across 144 findings and significantly improved radiologist interpretation performance. • The assistance of the comprehensive deep learning model significantly reduced the time required for radiologists to interpret computed tomography scans of the brain. : :

2 MIN READ
Clinical research Artificial intelligence Annalise CTB

Analysis of Line and Tube Detection Performance of a Chest X-ray Deep Learning Model to Evaluate Hidden Stratification

Diagnostics 2023, 13(14), 2317; Published: 9 July 2023 Authors: Cyril H. M. Tang, Jarrel C. Y. Seah, Hassan K. Ahmad, Michael R. Milne, Jeffrey B. Wardman, Quinlan D. Buchlak, Nazanin Esmaili, John F. Lambert, Catherine M. Jones Abstract: This retrospective case-control study evaluated the diagnostic performance of a commercially available chest radiography deep convolutional neural network (DCNN) in identifying the presence and position of central venous catheters, enteric tubes, and endotracheal tubes, in addition to a subgroup analysis of different types of lines/tubes. A held-out test dataset of 2568 studies was sourced from community radiology clinics and hospitals in Australia and the USA, and was then ground-truth labelled for the presence, position, and type of line or tube from the consensus of a thoracic specialist radiologist and an intensive care clinician. DCNN model performance for identifying and assessing the positioning of central venous catheters, enteric tubes, and endotracheal tubes over the entire dataset, as well as within each subgroup, was evaluated. The area under the receiver operating characteristic curve (AUC) was assessed. The DCNN algorithm displayed high performance in detecting the presence of lines and tubes in the test dataset with AUCs > 0.99, and good position classification performance over a subpopulation of ground truth positive cases with AUCs of 0.86–0.91. The subgroup analysis showed that model performance was robust across the various subtypes of lines or tubes, although position classification performance of peripherally inserted central catheters was relatively lower. Our findings indicated that the DCNN algorithm performed well in the detection and position classification of lines and tubes, supporting its use as an assistant for clinicians. Further work is required to evaluate performance in rarer scenarios, as well as in less common subgroups. : : :

2 MIN READ
Annalise CXR Clinical research Artificial intelligence

Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review

Diagnostics 2023, 13(4), 743; Published: 15 February 2023 Authors: Hassan K. Ahmad, Michael R. Milne, Quinlan D. Buchlak, Nalan Ektas, Georgina Sanderson, Hadi Chamtie, Sajith Karunasena, Jason Chiang, Xavier Holt, Cyril H. M. Tang, Jarrel C. Y. Seah, Georgina Bottrell, Nazanin Esmaili, Peter Brotchie, Catherine Jones Abstract: Limitations of the chest X-ray (CXR) have resulted in attempts to create machine learning systems to assist clinicians and improve interpretation accuracy. An understanding of the capabilities and limitations of modern machine learning systems is necessary for clinicians as these tools begin to permeate practice. This systematic review aimed to provide an overview of machine learning applications designed to facilitate CXR interpretation. A systematic search strategy was executed to identify research into machine learning algorithms capable of detecting >2 radiographic findings on CXRs published between January 2020 and September 2022. Model details and study characteristics, including risk of bias and quality, were summarized. Initially, 2248 articles were retrieved, with 46 included in the final review. Published models demonstrated strong standalone performance and were typically as accurate, or more accurate, than radiologists or non-radiologist clinicians. Multiple studies demonstrated an improvement in the clinical finding classification performance of clinicians when models acted as a diagnostic assistance device. Device performance was compared with that of clinicians in 30% of studies, while effects on clinical perception and diagnosis were evaluated in 19%. Only one study was prospectively run. On average, 128,662 images were used to train and validate models. Most classified less than eight clinical findings, while the three most comprehensive models classified 54, 72, and 124 findings. This review suggests that machine learning devices designed to facilitate CXR interpretation perform strongly, improve the detection performance of clinicians, and improve the efficiency of radiology workflow. Several limitations were identified, and clinician involvement and expertise will be key to driving the safe implementation of quality CXR machine learning systems. : : :

2 MIN READ
Artificial intelligence Annalise CXR

Charting the potential of brain computed tomography deep learning systems

Journal of Clinical Neuroscience, open access. May 2022. https://doi.org/10.1016/j.jocn.2022.03.014 Authors: Quinlan D.Buchlak, Michael R.Milne, Jarrel Seah, Andrew Johnson, Gihan Samarasinghe, Ben Hachey, Nazanin Esmaili, Aengus Tran, Jean-Christophe Leveque, Farrokh Farrokhi, Tony Goldschlager, Simon Edelstein, Peter Brotchie Abstract: Brain computed tomography (CTB) scans are widely used to evaluate intracranial pathology. The implementation and adoption of CTB has led to clinical improvements. However, interpretation errors occur and may have substantial morbidity and mortality implications for patients. Deep learning has shown promise for facilitating improved diagnostic accuracy and triage. This research charts the potential of deep learning applied to the analysis of CTB scans. It draws on the experience of practicing clinicians and technologists involved in development and implementation of deep learning-based clinical decision support systems. We consider the past, present and future of the CTB, along with limitations of existing systems as well as untapped beneficial use cases. Implementing deep learning CTB interpretation systems and effectively navigating development and implementation risks can deliver many benefits to clinicians and patients, ultimately improving efficiency and safety in healthcare. : : :

2 MIN READ
Thought leadership Annalise CTB

Diagnostic accuracy of a commercially available deep learning algorithm in supine chest radiographs following trauma

BJR. First published online 18 Mar 2022. Authors: Jacob Gipson, Victor Tang, Jarrel Seah, Helen Kavnoudias, Adil Zia, Robin Lee, Biswadev Mitra and Warren Clements Abstract: Objectives: Trauma chest radiographs may contain subtle and time-critical pathology. Artificial intelligence (AI) may aid in accurate reporting, timely identification and worklist prioritisation. However, few AI programs have been externally validated. This study aimed to evaluate the performance of a commercially available deep convolutional neural network – Annalise CXR V1.2 (Annalise.ai)- for detection of traumatic injuries on supine chest radiographs. Methods: Chest radiographs with a CT performed within 24 h in the setting of trauma were retrospectively identified at a level one adult trauma centre between January 2009 and June 2019. Annalise.ai assessment of the chest radiograph was compared to the radiologist report of the chest radiograph. Contemporaneous CT report was taken as the ground truth. Agreement with CT was measured using Cohen’s κ and sensitivity/specificity for both AI and radiologists were calculated. Results: There were 1404 cases identified with a median age of 52 (IQR 33–69) years, 949 male. AI demonstrated superior performance compared to radiologists in identifying pneumothorax (p = 0.007) and segmental collapse (p = 0.012) on chest radiograph. Radiologists performed better than AI for clavicle fracture (p = 0.002), humerus fracture (p < 0.0015) and scapula fracture (p = 0.014). No statistical difference was found for identification of rib fractures and pneumomediastinum.

2 MIN READ
Clinical research Annalise CXR

Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study

BMJ Open. First published December 20, 2021 Authors: Catherine M Jones, Luke Danaher, Michael R Milne, Cyril Tang, Jarrel Seah, Luke Oakden-Rayner, Andrew Johnson, Quinlan D Buchlak, Nazanin Esmaili Abstract: Objectives: Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists. Design: This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting. Setting: The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020.

2 MIN READ
Clinical research Annalise CXR
Load More