Transformer-Evolution-Paper

README
数学符号
Act
- A survey on recently proposed activation functions for Deep Learning
Arch
FFN
Head
- Multi-Head Attention Collaborate Instead of Concatenate
- Fast Transformer Decoding: One Write-Head is All You Need
Memory
MHA
Normalize_And_Residual
Pe
Pretrain
Softmax
- Transformer with a Mixture of Gaussian Keys
- Normalized Attention Without Probability Cage
Others
LongConv
Rnn
CrossAttention
- Neural Machine Translation in Linear Time
Inference
- Extrapolation
  Parallel Context Windows for Large Language Models
  Structured Prompting: Scaling In-Context Learning to 1,000 Examples
  Naive Bayes-based Context Extension
Peft
- Parameter-Efficient Fine-Tuning without Introducing New Latency
- Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
LLM
- LLM Details Summary
- What Language Model to Train if You Have One Million GPU Hours?

Powered by GitBook

On this page

CrossAttention

Neural Machine Translation in Linear Time

PreviousQuasi-recurrent neural networks NextNeural Machine Translation in Linear Time

Last updated 2 years ago