Transformer-Evolution-Paper

README
数学符号
Act
- A survey on recently proposed activation functions for Deep Learning
Arch
FFN
Head
- Multi-Head Attention Collaborate Instead of Concatenate
- Fast Transformer Decoding: One Write-Head is All You Need
Memory
MHA
Normalize_And_Residual
Pe
Pretrain
Softmax
- Transformer with a Mixture of Gaussian Keys
- Normalized Attention Without Probability Cage
Others
LongConv
Rnn
CrossAttention
- Neural Machine Translation in Linear Time
Inference
- Extrapolation
Peft
- Parameter-Efficient Fine-Tuning without Introducing New Latency
- Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
LLM
- LLM Details Summary
- What Language Model to Train if You Have One Million GPU Hours?

Powered by GitBook

On this page

整体思路以及计算方式
时间复杂度
训练以及loss
代码
实验以及适用场景
细节
简评

MHA

RightProduct

Transformer with Fourier Integral Attentions

PreviousAn Attention Free Transformer NextLinear Complexity Randomized Self-attention Mechanism

Last updated 2 years ago