marmot_angora.d

Recent research on language models has aimed to increase the maximum allowable context length of the underlying model. But, how can we enable an LLM to handle longer inputs? One way is through the use of ALiBi… Vanilla position embeddings. Decoder-only transformer architectures create an input sequence of tokens by first tokenizing the raw text and looking up the embedding (each token in the tokenizer’s vocabulary has a unique embedding) for each token. Then, we add a positional embedding to each token embedding, which injects positional info into the embedding of each token in the sequence. Why is this necessary? We should recall that the self-attention operation is agnostic to the position of a token in a sequence! The only way to consider the position of each token when computing self-attention is to inject such information into each token’s embedding or into the attention computation itself. ALiBi > position embeddings. Attention with Linear Biases (ALiBi) gets rid of positional embeddings altogether. Instead, positional information is injected into the transformer as part of the self-attention operation via adding a penalty to the key-query attention score. In particular, we should recall that self-attention computes an attention score between each pair of tokens within a sequence. ALiBi operates by adding a static, non-learned bias to this score that is proportional to the distance between the pair of tokens. The result. Such an approach is impactful because it depends on the pairwise distance between tokens rather than the absolute positions of tokens within a sequence. This quantity is less dependent upon the length of the underlying sequence and allows ALiBi to generalize much better to sequences that are longer than those seen during training. For example, MPT-7B (which uses ALiBi) can be fine-tuned on sequences as long as 64K and reasonably extrapolate to much longer sequences of 84K tokens.