Integrated image-based deep learning and language models for primary diabetes care
Jiajia LiMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaZhouyu GuanShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaJing WangDepartment of Ophthalmology, Huadong Sanatorium, Wuxi, ChinaCarol Y. CheungDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaYingfeng ZhengState Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, ChinaLee‐Ling LimDepartment of Medicine, Faculty of Medicine, University of Malaya, Kuala Lumpur, MalaysiaCynthia Ciwei LimDepartment of Renal Medicine, Singapore General Hospital, SingHealth-Duke Academic Medical Centre, Singapore, SingaporePaisan RuamviboonsukFaculty of Medicine, Department of Ophthalmology, Rajavithi Hospital, College of Medicine, Rangsit University, Bangkok, ThailandRajiv RamanShri Bhagwan Mahavir Vitreoretinal Services, Medical Research Foundation, Sankara Nethralaya, Chennai, IndiaLeonor CorsinoDepartment of Medicine, Division of Endocrinology, Metabolism and Nutrition, and Department of Population Health Sciences, Duke University School of Medicine, Durham, NC, USAJustin B. Echouffo‐TcheuguiDepartment of Medicine, Division of Endocrinology, Diabetes and Metabolism, Johns Hopkins School of Medicine, Baltimore, MD, USAAndrea O. Y. LukAsia Diabetes Foundation, Hong Kong Special Administrative Region, ChinaLi Jia ChenDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaXiaodong SunDepartment of Ophthalmology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaHaslina HamzahSingapore Eye Research Institute, Singapore National Eye Centre, Singapore, SingaporeQiang WuDepartment of Ophthalmology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaXiangning WangDepartment of Ophthalmology, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaRuhan LiuMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaYa Xing WangBeijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Medical University, Beijing Ophthalmology and Visual Sciences Key Laboratory, Beijing, ChinaTing‐Li ChenDepartment of Ophthalmology, Huadong Sanatorium, Wuxi, ChinaXiao ZhangXiaolong YangDepartment of Ophthalmology, Huadong Sanatorium, Wuxi, ChinaJun YinShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaJing WanDepartment of Endocrinology and Metabolism, Shanghai Eighth People's Hospital, Shanghai, ChinaDu WeiDepartment of Endocrinology and Metabolism, Shanghai Eighth People's Hospital, Shanghai, ChinaTen Cheer QuekSingapore Eye Research Institute, Singapore National Eye Centre, Singapore, SingaporeJocelyn Hui Lin GohSingapore Eye Research Institute, Singapore National Eye Centre, Singapore, SingaporeDawei YangDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaXiaoyan HuDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaTruong NguyenDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaSimon SzetoDepartment of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, ChinaPeranut ChotcomwongseFaculty of Medicine, Department of Ophthalmology, Rajavithi Hospital, College of Medicine, Rangsit University, Bangkok, ThailandRachid MalekDepartment of Internal Medicine, Setif University Ferhat Abbas, Setif, AlgeriaNargiza NormatovaOphthalmology Department at Tashkent Advanced Training Institute for Doctors, Tashkent, UzbekistanNilufar IbragimovaRamyaa SrinivasanShri Bhagwan Mahavir Vitreoretinal Services, Medical Research Foundation, Sankara Nethralaya, Chennai, IndiaPingting ZhongState Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, ChinaWenyong HuangState Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, ChinaChenxin DengDepartment of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, ChinaLei RuanDepartment of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, ChinaCuntai ZhangDepartment of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, ChinaChenxi ZhangDepartment of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, ChinaYan ZhouDepartment of Ophthalmology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, ChinaChan WuDepartment of Ophthalmology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, ChinaRongping DaiDepartment of Ophthalmology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, ChinaSky Wei Chee KohNational University Polyclinics, National University Health System, Department of Family Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, SingaporeAdina AbdullahDepartment of Primary Care Medicine, Faculty of Medicine, Universiti Malaya, Kuala Lumpur, MalaysiaNicholas Ken Yoong HeeDepartment of Medicine, University Malaya Medical Centre, Kuala Lumpur, MalaysiaHong Chang TanDepartment of Endocrinology, Singapore General Hospital, Singapore, SingaporeZhong Hong LiewDepartment of Renal Medicine, Singapore General Hospital, SingHealth-Duke Academic Medical Centre, Singapore, SingaporeCarolyn Shan‐Yeu TienSingapore General HospitalShih Ling KaoDepartment of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, SingaporeAmanda Yuan Ling LimDepartment of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, SingaporeShao Feng MokDepartment of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, SingaporeLina SunDepartment of Internal Medicine, Huadong Sanatorium, Wuxi, ChinaJing GuDepartment of Internal Medicine, Huadong Sanatorium, Wuxi, ChinaLiang WuShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaTingyao LiMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaDi ChengShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaZheyuan WangMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaYiming QinMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaLing DaiMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaZiyao MengMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaJia ShuMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaYuwei LuShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaNan JiangMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaTingting HuShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaShan HuangMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaGengyou HuangMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaShujie YuShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaDan LiuShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaWeizhi MaInstitute for AI Industry Research, Tsinghua University, Beijing, ChinaMinyi GuoShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaXinping GuanDepartment of Automation and the Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai Jiao Tong University, Shanghai, ChinaXiaokang YangDepartment of Ophthalmology, Huadong Sanatorium, Wuxi, ChinaCovadonga BascaránInternational Centre for Eye Health, London School of Hygiene and Tropical Medicine, University of London, London, UKCharles R ClelandInternational Centre for Eye Health, London School of Hygiene and Tropical Medicine, University of London, London, UKYuqian BaoShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, ChinaElif I. EkinciAustralian Centre for Accelerating Diabetes Innovations, The University of Melbourne, Parkville, Victoria, AustraliaAlicia J. JenkinsAustralian Centre for Accelerating Diabetes Innovations, The University of Melbourne, Parkville, Victoria, AustraliaJuliana C.N. ChanAsia Diabetes Foundation, Hong Kong Special Administrative Region, ChinaYong Mong BeeDepartment of Endocrinology, Singapore General Hospital, Singapore, SingaporeSobha SivaprasadNIHR Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UKJonathan E. ShawDepartment of Medicine, The University of Melbourne (Austin Health), Melbourne, Victoria, AustraliaRafael SimóCentro de Investigación Biomédica en Red de Diabetes y Enfermedades Metabólicas Asociadas, Instituto de Salud Carlos III, Madrid, SpainPearse A. KeaneInstitute of Ophthalmology, University College London, London, UKChing‐Yu ChengCenter for Innovation and Precision Eye Health and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, SingaporeGavin Siew Wei TanSingapore Eye Research Institute, Singapore National Eye Centre, Singapore, SingaporeWeiping JiaShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China. [email protected]Yih Chung ThamCenter for Innovation and Precision Eye Health and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore. [email protected]Huating LiShanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China. [email protected]Bin ShengMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China. [email protected]Tien Yin WongBeijing Tsinghua Changgung Hospital, Beijing, China. [email protected]
ABI
Abstract
Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.
Topics
Identifiers
Citations and references
Cited by 071 references