位置编码

Transformer 式

{\begin{aligned} p_{k, 2 i} & = \sin (k / 10000^{2 i / d}) \\ p_{k, 2 i + 1} & = \cos (k / 10000^{2 i / d}) \end{aligned}

具有外推性

{\begin{cases} q_{i} = (x_{i} + p_{i}) W_{Q} \\ k_{j} = (x_{j} + p_{j}) W_{K} \\ v_{j} = (x_{j} + p_{j}) W_{V} \\ a_{i, j} = s o f t m a x (q_{i} k_{j}^{⊤}) \\ o_{i} = \sum_{j} a_{i, j} v_{j} \end{cases}

展开 $q_{i} k_{j}^{T}$ ：

q_{i} k_{j}^{⊤} = (x_{i} + p_{i}) W_{Q} W_{K}^{⊤} (x_{j} + p_{j})^{⊤} = (x_{i} W_{Q} + p_{i} W_{Q}) (W_{K}^{⊤} x_{j}^{⊤} + W_{K}^{⊤} p_{j}^{⊤})