> For the complete documentation index, see [llms.txt](https://doraemonzzz.gitbook.io/transformer_evolution_paper/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doraemonzzz.gitbook.io/transformer_evolution_paper/normalize_and_residual/002.md).

# Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

论文地址：

* <https://arxiv.org/abs/2002.10444>

## 整体思路以及计算方式

思路和ReZero类似，用加权残差取代Normalize，唯一的区别是这篇讨论的是Batch Normalize，而ReZero讨论的是Layer Normalize：

$$
\mathbf {x}*{i+1}=\mathbf {x}*{i}+\alpha\_{i} F\left(\mathbf {x}\_{i}\right)
$$

后续部分从略。