What is Auto regressive decoding in LLMs

Auto-regressive decoding is the fundamental process by which most large language models (LLMs) generate text, one token at a time, in a sequential manner. The core idea is that the model predicts the next token based on all the tokens that have been generated so far, including the initial input prompt.

Here's a breakdown of how it works:

The Process:

Input: The process starts with an input prompt, which is a sequence of tokens.

Encoding: The LLM first processes this input prompt, typically by converting the tokens into numerical representations called embeddings.

First Token Prediction: Based on the encoded prompt, the model predicts the probability distribution over its entire vocabulary for the next token.

Token Sampling/Selection: A decoding strategy is then used to select the next token from this probability distribution. Common strategies include:

Greedy Decoding: Simply selecting the token with the highest probability. This is fast but can lead to repetitive or suboptimal outputs.

Sampling: Randomly selecting a token based on its probability. This introduces more diversity but can also lead to less coherent outputs.

Beam Search: Keeping track of multiple promising candidate sequences (beams) and expanding them at each step. This often yields better quality text than greedy decoding but is more computationally expensive.

Appending the Token: The selected token is appended to the currently generated sequence.

Iteration: The model then takes the original prompt plus the newly generated token as the new input and repeats steps 3-5 to predict the next token.

Stopping Condition: This process continues until a predefined stopping condition is met, such as reaching a maximum sequence length or generating a special "end-of-sequence" token.

Output: The final sequence of generated tokens is then converted back into human-readable text.

Why is it called "Auto-regressive"?

The term "auto-regressive" comes from statistics and signal processing. In this context, it means that the model's output at each step is dependent on its own previous outputs. The model "regresses" on its own generated history to predict the future.

Key Characteristics:

Sequential Generation: Tokens are generated one after the other. This inherent sequential nature can be a bottleneck for inference speed, especially for long sequences.

Context Dependency: Each predicted token is conditioned on the entire preceding context. This allows the model to maintain coherence and relevance in its generated text.

Probability Distribution: At each step, the model outputs a probability distribution over the vocabulary, allowing for different decoding strategies to influence the final output.

Implications:

Inference Speed: The sequential nature of auto-regressive decoding is a primary factor contributing to the latency of LLM inference. Generating longer sequences requires more steps.

Computational Cost: Each decoding step involves a forward pass through the model, which can be computationally intensive for large models.

Decoding Strategy Impact: The choice of decoding strategy significantly affects the quality, diversity, and coherence of the generated text, as well as the inference speed.

In summary, auto-regressive decoding is the step-by-step process of generating text by predicting one token at a time, with each prediction being conditioned on the previously generated tokens and the initial input. It's a fundamental mechanism behind the impressive text generation capabilities of modern LLMs.

References 

Gemini