Table 2 Hyperparameters of the ERNIE model
Parameter | Value |
|---|---|
Attention_probs_dropout_prob | 0.1 |
Activation Function | Relu |
Hidden_dropout_prob | 0.1 |
Hidden_size | 768 |
Initializer_range | 0.02 |
Max_position_embeddings | 513 |
Num_attention_heads | 12 |
Num_hidden_layers | 12 |
Type_vocab_size | 2 |
Vocab_size | 18,000 |