Explain the Transformer Architecture and Multi-Head Self-Attention in LLMs

The Transformer architecture, introduced in the seminal paper "Attention Is All You Need" (Vaswani et al., 2017), replaced recurrent and convolutional layers with Self-Attention mechanisms, enabling massive parallel processing across sequence tokens.

Core Mathematical Formulation: Scaled Dot-Product Attention

Given an input sequence mapped to Query (Q), Key (K), and Value (V) matrices of dimension $d_k$:

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

Why Scaled Dot-Product?

• Dot Product $Q \cdot K^T$: Computes the pairwise similarity score between every query token and all key tokens in the context window.
• Scaling Factor $1 / \sqrt{d_k}$: For large values of $d_k$, dot products grow large in magnitude, pushing the softmax function into regions with extremely small gradients (vanishing gradient problem). Dividing by $\sqrt{d_k}$ stabilizes numerical training.
• Softmax: Normalizes similarities into attention weights between 0 and 1 that sum to 1.
• Weighted Sum with $V$: Aggregates value vectors weighted by attention relevance.

Multi-Head Attention Mechanism

Rather than performing a single attention function, Multi-Head Attention projects Queries, Keys, and Values $h$ times with distinct, lea able linear projections:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) * W^O
where head_i = Attention(Q * W_i^Q, K * W_i^K, V * W_i^V)

This allows the model to jointly attend to information from different representation subspaces at different positions (e.g., grammatical dependencies, semantic references, long-range factual associations).