Multilayer Networks as Composed Linear Transformations

• Write a dense (fully-connected) layer's forward computation as a matrix-vector product plus a bias, exactly matching the linear-transformation view of matrices already covered in this curriculum.
• Explain precisely why stacking dense layers without a nonlinear activation between them collapses into one equivalent linear layer, and why this makes the activation function, not the number of layers, the true source of a network's expressive power.
• Trace a full forward pass, layer by layer, through a small network with concrete numbers.
• State this discipline's scope and its exact starting point relative to ai-theory/machine-learning.

ai-theory/machine-learning ended, deliberately, at from-perceptrons-to-neural-networks: stacking perceptron-like units into layers, with a smooth activation function replacing the hard threshold, breaks the single linear boundary that limited every classifier in that discipline. That concept stopped exactly there, naming backpropagation, deep architectures, and large-scale training as the entire, separate scope of this discipline. This concept is the direct continuation: it gives the layered network a precise, matrix-based description before anything is trained.

This matters immediately for two reasons. First, foundations/mathematics-for-computing already proved that a matrix is nothing more than a linear transformation and that composing linear transformations means multiplying their matrices — a dense layer's weight matrix is a real instance of exactly that object, not a new mathematical idea. Second, framing the forward pass as a sequence of matrix operations is what makes it fast: modern deep learning hardware (GPUs, covered later in this discipline's CNN cluster alongside computer/computer-architecture) is built to execute exactly this kind of large, regular matrix multiplication efficiently.

A dense layer as a matrix-vector product

A dense layer with input vector x (length n), weight matrix W (shape m × n), and bias vector b (length m) computes a pre-activation vector:

text
z = Wx + b

Each of the m output entries of z is a weighted sum (dot product) of all n inputs, plus that output unit's own bias — exactly the "linear combination of inputs" already familiar from linear regression and the perceptron, just computed for m output units simultaneously using matrix notation. W here is precisely the matrix that matrices-as-linear-transformations (foundations/mathematics-for-computing) already characterized: it maps a vector in one space (dimension n) to a vector in another (dimension m).

Why linear layers alone cannot compose into anything new

Suppose two dense layers were stacked with no nonlinearity in between: z₁ = W₁x + b₁, then z₂ = W₂z₁ + b₂. Substituting:

text
z₂ = W₂(W₁x + b₁) + b₂
   = (W₂W₁)x + (W₂b₁ + b₂)
   = W'x + b'

where W' = W₂W₁ and b' = W₂b₁ + b₂ — a single new weight matrix and bias, exactly matrix-operations' composition-of-linear-maps fact applied directly. No matter how many purely linear layers are stacked, the entire composition is algebraically identical to one linear layer. This is the precise, provable version of the informal claim already made in from-perceptrons-to-neural-networks: nonlinearity between layers, not depth by itself, is the entire source of a multilayer network's extra expressive power.

The activation function breaks the collapse

Inserting a nonlinear activation function φ (sigmoid, tanh, or — far more common in modern practice — ReLU, φ(z) = max(0, z)) after each linear step changes the picture entirely:

text
h₁ = φ(W₁x + b₁)
h₂ = φ(W₂h₁ + b₂)

Because φ is applied element-wise and is not itself a linear function, there is no matrix W' for which W'x + b' = φ(W₂φ(W₁x + b₁) + b₂) in general. This is the mathematical content behind the entire idea of a "hidden layer": each layer's nonlinear output becomes the next layer's input, and the whole network — however many layers deep — is one composed, genuinely nonlinear function of x.

Example 1: A hand-traced two-layer forward pass

Consider a network with input x = (1, 2), one hidden layer of 2 ReLU units with weight matrix W₁ = [[1, -1], [0.5, 0.5]] and zero bias, and an output layer with weight vector w₂ = (1, 1) and zero bias:

text
Hidden pre-activation: z₁ = W₁x = (1(1) + (-1)(2), 0.5(1) + 0.5(2)) = (-1, 1.5)
Hidden activation:     h₁ = ReLU(z₁) = (max(0,-1), max(0,1.5)) = (0, 1.5)
Output:                ŷ = w₂ · h₁ = 1(0) + 1(1.5) = 1.5

Every step is one matrix-vector product followed by one element-wise nonlinearity — the entire forward pass of any feedforward network, regardless of depth, is this pattern repeated layer after layer.

Example 2: The collapse made concrete

Take the same W₁ and w₂ as above, but remove the ReLU. Composing algebraically: w₂ᵀW₁ = 1(1) + 1(0.5), 1(-1) + 1(0.5) = (1.5, -0.5). Checking against x = (1, 2): (1.5)(1) + (-0.5)(2) = 1.5 - 1 = 0.5. Directly: w₂ · (W₁x) = w₂ · (-1, 1.5) = -1 + 1.5 = 0.5. Both routes give the same answer, 0.5 — confirming that without the ReLU, the two-layer network is exactly equivalent to a single linear layer with weight vector (1.5, -0.5), for every possible input, not just this one.

• "A network with more layers is automatically more expressive." Only true when a nonlinear activation separates the layers — Example 2 shows a concrete case where two purely linear layers reduce, exactly, to one. Depth without nonlinearity buys nothing.
• "The weight matrix W is just a table of numbers, unrelated to the linear algebra already covered." W is precisely the object matrices-as-linear-transformations described: a matrix that maps one vector space to another. Nothing about how a dense layer computes its output is a new mathematical primitive.
• "This concept already covers neural networks, so machine-learning's stopping point was redundant." from-perceptrons-to-neural-networks covered only the minimal representational claim (nonlinearity breaks the linear ceiling, demonstrated on XOR). This concept restates that claim with precise matrix notation as the deliberate on-ramp into this discipline — training multilayer networks at scale (backpropagation onward) starts with the very next concept.

A dense layer computes z = Wx + b, exactly the linear-transformation-plus-composition machinery already proven in this curriculum's linear algebra discipline; stacking such layers without a nonlinearity between them algebraically collapses into one equivalent linear layer, provable directly by matrix multiplication. Inserting a nonlinear activation function after each linear step is what prevents this collapse and gives a multilayer network its real expressive power — the exact mechanism from-perceptrons-to-neural-networks gestured at, now made precise as the entry point into this discipline's coverage of training and scaling these networks.