Статья

Developing Pretrained Language Models for Turkish Biomedical Domain

Hazal TürkmenEge University,Faculty of Engineering,Computer Engineering Department,İIzmir,TurkiyeOğuz DikenelliEge University,Faculty of Engineering,Computer Engineering Department,İIzmir,TurkiyeCenk EraslanEge University,Faculty of Medicine,Radiology Department,İIzmir,TurkiyeMehmet Cem ÇallıEge University,Faculty of Medicine,Radiology Department,İIzmir,TurkiyeSüha Süreyya ÖzbekEge University,Faculty of Medicine,Radiology Department,İIzmir,Turkiye

2022en

ABI

Аннотация

Pretrained language models elevated with in-domain corpora show impressive results in biomedicine and clinical NLP tasks in English. However, there is minimal work in low-resource languages. This work introduces the BioBERTurk family, three pretrained models in Turkish for biomedicine. To evaluate models, we also introduce a labeled dataset to classify radiology reports of CT exams. Our first model was initialized from BERTurk and pretrained with biomedical corpus. The second model again continues to pretrain the general BERT model with a corpus of Ph.D. theses on radiology to test the effect of the task-related text. The final model combines radiology and biomedicine corpora with the corpus of BERTurk and pretrained a BERT model from scratch. F-scores of our models in the radiology resort classification are 92.99, 92.75, and 89.49 respectively. As far as we know, this is the first model that evaluates the effect of small size in-domain corpus in pretraining from scratch.

Перевод пока недоступен

Идентификаторы

DOI: 10.1109/ichi54592.2022.00117

Цитирования и источники

Цитирований: 6Использованных источников: 0

Показатели — AkademScholar