Table 1 The recent pre-trained transformers models
From: Adapting transformer-based language models for heart disease detection and risk factors extraction
Model | Pre-trained dataset | The training weights | References |
|---|---|---|---|
BERT model | Book Corpus Wikipedia | – | [33] |
BioBERT model | PubMed PubMed Central articles | BERT weights | [33] |
BlueBERT model | PubMed PMC MIMIC III data | BERT weights | [75] |
ClinicalBERT model | MIMIC III data | BioBERT weights | [34] |
SciBERT model | Semantic Scholar data | BERT weights | [76] |
PubMedBERT model | PubMed PMC data | BERT weights | [77] |
Bio-lm model | PMC PubMed MIMIC III | BERT weights | [78] |