# Pretrain

- [XLNet Generalized Autoregressive Pretraining for Language Understanding](/transformer_evolution_paper/pretrain/001.md)
- [Transcormer Transformer for Sentence Scoring with Sliding Language Modeling](/transformer_evolution_paper/pretrain/002.md)
- [Optimus Organizing Sentences via Pre-trained Modeling of a Latent Space](/transformer_evolution_paper/pretrain/003.md)
- [ELECTRA Pre-training Text Encoders as Discriminators Rather Than Generators](/transformer_evolution_paper/pretrain/004.md)
- [Cramming: Training a Language Model on a Single GPU in One Day](/transformer_evolution_paper/pretrain/005.md)
