#ffn

The Activation Is a Gate

Every modern open-weight LLM replaced its FFN activation function with SwiGLU. That swap added a third weight matrix, changed the hidden dimension arithmetic, and altered how information flows through every feed-forward layer. It is not a cosmetic improvement. It is a structural decision made once at training time that every inference system running the model must pay for.