> For the complete documentation index, see [llms.txt](https://doraemonzzz.gitbook.io/transformer_evolution_paper/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doraemonzzz.gitbook.io/transformer_evolution_paper/inference/extrapolation/003.md).

# Naive Bayes-based Context Extension

论文地址：

* <https://spaces.ac.cn/archives/9617>

## 整体思路以及计算方式

苏神提出的方法，注意思路基于朴素贝叶斯，问题的描述为根据问题$$S\_1,\ldots, S\_n$$生成$$T$$，即估计$$p\left(T |S\_1, S\_2, \cdots, S\_n\right)$$，根据贝叶斯公式可得：

$$
p\left(T| S\_1, S\_2, \cdots, S\_n\right) \propto p\left(S\_1, S\_2, \cdots, S\_n | T\right) p(T)
$$

根据朴素贝叶斯假设可得：

$$
p\left(S\_1, S\_2, \cdots, S\_n |T\right) = \prod\_{i=1}^n p\left(S\_i| T\right) =\prod\_{i=1}^n \frac{p(T|S\_i)p(S\_i)}{p(T)}
$$

即：

$$
p\left(T| S\_1, S\_2, \cdots, S\_n\right) \propto \frac{\prod\_{i=1}^n p\left(T| S\_i\right)}{p^{n-1}(T)}
$$

所以可以根据右式进行采样。转换为对数概率情形可得：

$$
\log p\left(T| S\_1, S\_2, \cdots, S\_n\right)= {\sum\_{i=1}^n p\left(T| S\_i\right)} -(n-1) {p(T)} + C=n \overline{p\left(T| S\right)}-(n-1)p(T)+C \ \overline{p\left(T| S\right)}=\frac{ {\sum\_{i=1}^n p\left(T| S\_i\right)}}{n}
$$

然后苏神引入超参数$$\beta$$，上式变为：

$$
\log p\left(T| S\_1, S\_2, \cdots, S\_n\right)=\beta \overline{p\left(T| S\right)}-(\beta-1)p(T)
$$

## 代码

* <https://github.com/bojone/NBCE>

## 细节

实现时，将$$\varnothing, S\_1, S\_2, \ldots, S\_n$$分别作为模型的输入得到$$n+1$$个结果，然后基于上述方法进行采样即可。