Transformer-Evolution-Paper

README
数学符号
Act
- A survey on recently proposed activation functions for Deep Learning
Arch
FFN
Head
- Multi-Head Attention Collaborate Instead of Concatenate
- Fast Transformer Decoding: One Write-Head is All You Need
Memory
MHA
Normalize_And_Residual
Pe
Pretrain
Softmax
- Transformer with a Mixture of Gaussian Keys
- Normalized Attention Without Probability Cage
Others
LongConv
Rnn
CrossAttention
- Neural Machine Translation in Linear Time
Inference
- Extrapolation
Peft
- Parameter-Efficient Fine-Tuning without Introducing New Latency
- Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
LLM
- LLM Details Summary
- What Language Model to Train if You Have One Million GPU Hours?

Powered by GitBook

On this page

MHA

FFT LocalGlobal MatrixMethod RightProduct SparseOrLowRank Others

PreviousAdapting Language Models to Compress Contexts NextFFT

Last updated 2 years ago