Transformer-Evolution-Paper

⌘Ctrlk

FFN

Large Memory Layers with Product Keys Transformer Feed-Forward Layers Are Key-Value Memories GLU Variants Improve Transformer Simple Recurrence Improves Masked Language Models Pay Attention to MLPs S2-MLP Spatial-Shift MLP Architecture for Vision S2-MLPv2 Improved Spatial-Shift MLP Architecture for Vision HyperMixer An MLP-based Green AI Alternative to Transformers DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling & DeLighT: Deep and Light-weight Transformer When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

PreviousGeneralization through Memorization: Nearest Neighbor Language Models NextLarge Memory Layers with Product Keys

Last updated 2 years ago