Regularization in Deep Networks: Dropout and Weight Decay

• Explain why deep networks, with millions of parameters, are especially prone to overfitting in the sense already formalized by the bias-variance tradeoff and the VC dimension.
• Show that weight decay is the exact ridge (L2) penalty already covered, applied to a network's weights during gradient-based training.
• Explain the dropout mechanism precisely: what happens during training versus at test time, and why the difference matters.
• Explain, conceptually, why dropout has no direct analogue among the classical models ai-theory/machine-learning covered.

the-bias-variance-tradeoff, overfitting-and-the-vc-dimension, and regularization-ridge-and-lasso (ai-theory/machine-learning) established the general framework this concept inherits without re-deriving: a model flexible enough to fit its training data arbitrarily well is also flexible enough to fit noise, and some form of penalty or constraint on that flexibility is needed to generalize. A deep network with millions of weights is, by the VC-dimension argument already covered, an enormously high-capacity model class — routinely capable of driving training loss to near zero while still generalizing well, but only when trained with real regularization in place. This concept covers two techniques: weight decay, which is literally ridge regression's penalty carried over unchanged, and dropout, a technique with no shallow-model precedent that exploits something only a network with many redundant hidden units can do.

Weight decay is ridge regression's penalty, unchanged

regularization-ridge-and-lasso added a penalty term λ‖θ‖² to the loss, shrinking every weight toward zero and trading a small increase in bias for a reduction in variance. Weight decay applies the identical penalty to a deep network's weights:

text
J(θ) = Loss(θ) + λ‖θ‖²

Differentiating this penalty term and folding it into the gradient descent update (with any of the optimizers from the previous concept) produces, at each step, an update that first shrinks every weight by a small multiplicative factor before applying the usual gradient step — hence "weight decay." Nothing about this is new machinery: it is the same L2 penalty, on the same total loss-plus-penalty objective, that ridge regression already used, now applied to a network whose "features" are the many learned hidden representations rather than a fixed, hand-chosen set.

Dropout: a regularizer with no shallow-model analogue

Dropout works differently. During each training step, every hidden unit is independently "dropped" (its output forced to zero) with some probability p (typically 0.3–0.5), and the surviving units' outputs are rescaled by 1/(1−p) so the expected total signal reaching the next layer stays the same. At test time, dropout is turned off entirely — every unit is used, with no dropping and no rescaling needed (having already been accounted for during training, in the common "inverted dropout" implementation).

The effect during training is that no hidden unit can rely on any specific set of other units being present on a given step — it must learn a useful, at least partially redundant contribution on its own, since any of its usual collaborators might be zeroed out at any moment. This directly discourages co-adaptation: units becoming so specialized to work only in combination with a few specific other units that the network as a whole becomes fragile and overfit to precise patterns in the training data. Dropout has no analogue among the classical models ai-theory/machine-learning covered specifically because those models (linear/logistic regression, SVMs, trees) do not have large numbers of redundant, co-adapting internal units in the first place — this is a regularization technique that only makes sense for architectures with many hidden units to begin with.

Why both, and why together

Weight decay and dropout address overfitting through genuinely different mechanisms — weight decay constrains the magnitude of every weight uniformly, while dropout prevents reliance on specific combinations of hidden units — and in practice both are frequently used together on the same network, since neither one subsumes the other.

Example 1: Weight decay's effect on a single gradient step

Consider one weight w = 2.0, a computed loss gradient ∂Loss/∂w = 0.5, learning rate α = 0.1, and weight decay coefficient λ = 0.01. The full gradient including the penalty term (∂(λw²)/∂w = 2λw) is:

text
∂J/∂w = ∂Loss/∂w + 2λw = 0.5 + 2(0.01)(2.0) = 0.5 + 0.04 = 0.54
w ← w − α(0.54) = 2.0 − 0.054 = 1.946

Compare to the update with no weight decay: w ← 2.0 − 0.1(0.5) = 1.95. The weight decay term pulled the update slightly further toward zero (1.946 versus 1.95) — a small, but consistent, extra shrinkage applied to every weight on every step, exactly the mechanism that keeps weights from growing unnecessarily large over the course of training.

Example 2: Dropout's expected-value bookkeeping

Consider a hidden layer with 4 units producing outputs (2, 4, 1, 3) before dropout, with dropout probability p = 0.5. Suppose units 2 and 4 (values 4 and 3) happen to be dropped on this particular training step. The surviving units are rescaled by 1/(1−p) = 1/0.5 = 2:

text
Raw survivors:    (2, 0, 1, 0)
Rescaled by 1/(1-p) = 2:  (4, 0, 2, 0)

The expected value of each unit's contribution, averaged over the random choice of which units survive, is unchanged: E[rescaled output] = p·0 + (1−p)·(original/( 1−p)) = original. This is exactly why the rescaling factor is 1/(1−p): it keeps the layer's expected total output the same during training (with dropout) as it will be at test time (without it), so no separate correction is needed at test time.

• "Dropout and weight decay do the same thing, just implemented differently." Weight decay shrinks every weight's magnitude uniformly; dropout randomly removes entire units, preventing over-reliance on specific combinations of units. A network can overfit in a way weight decay does not address (fragile co-adaptation between specific units) even while all its individual weights stay small.
• "Dropout should be applied at test time too, for consistency with training." Dropout is deliberately disabled at test time — the entire point of the 1/(1−p) rescaling during training is to make the trained network's expected behavior match its full, no-dropout behavior at test time, so the best prediction uses every unit, not a random subset.
• "A higher dropout probability p is always more effective at preventing overfitting." Too high a p removes so much of the network's capacity on each step that it can also underfit — dropout trades a strictly smaller effective network per step, and choosing p is a real bias-variance tradeoff (the-bias-variance-tradeoff), not a knob that only ever helps by being turned higher.

Weight decay is ridge regression's L2 penalty, carried over to deep networks unchanged: it shrinks every weight's magnitude a little on every gradient step. Dropout is a technique with no analogue among classical models — during training, it randomly zeroes out hidden units (rescaling survivors to preserve the expected signal) so no unit can rely on any specific combination of collaborators, directly discouraging fragile co-adaptation; at test time, dropout is disabled and the full network is used. Both address the same overfitting risk the-bias-variance-tradeoff and overfitting-and-the-vc-dimension already formalized, through genuinely different mechanisms, which is why they are frequently combined rather than treated as interchangeable.