GLU Improves Transformers

GLU Variants Improve Transformer

PDF Link

Summary

Pretty quick read, the basic idea is that GLU based activations when used in the the FFN layers of the transformer tend to reduce perplexity both on short and long horizon training runs.

Noam compares different variants of GLU and SwiGLU comes out on top both on stability on longer runs and across different tasks/datasets.

But what i loved about this is how it ends, and i quote:

We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

---

Based on some poking around on ChatGPT:

How it’s implemented in practice (why it can look like “only 2”)

Most implementations fuse W and V into a single linear layer that outputs 2*d_ff, then split:

• One big matrix W_in ∈ $R^{d_{model}\times (2d_{ff})}$
• Compute h = $x@W_{in}$
• Split h into a, b each of shape $d_{ff}$
• Output $\phi (a)\otimes b$
• Then apply $W2\in R^{d_{ff}\times d_{model}}$

So you’ll see “2 Linear layers” in code ($W_{in}$ and $W2$), but parameter-wise it’s still 3 logical matrices: W and V are just concatenated inside $W_{in}$.

---

On GLU

A standard Transformer FFN does:

$h=\varphi (x{W}_1+b_1),y=h{W}_2+b_2$

It’s expressive, but the “decision” (what to pass through) is entangled with the “content” (what the features are), because it’s all inside h .

GLU explicitly splits those roles:

$$\text{GLU}(x)=\underset{\text{gate in }(0,1)}{\underbrace{\sigma (x{W}_g)}}\otimes \underset{\text{value/content}}{\underbrace{(x{W}_v)}}$$

Two projections:

• Value branch ($W_v$): “what information do I have available?”
• Gate branch ($W_g$): “how much of each feature should I let through?”

Then you multiply them elementwise. That multiplication is the key: it creates feature-wise, input-dependent routing.

---

A Useful Intuition

• Activation-only FFN: “I’ll compute features and squish them.”

• GLU FFN: “I’ll compute features, and separately compute a controller that decides how much of each feature gets through.”

It’s the difference between “nonlinearity as a squashing function” vs “nonlinearity as conditional routing.”