From Perceptrons to Neural Networks

• Explain the perceptron's fundamental limitation — it can only represent linear decision boundaries — and why stacking perceptrons into layers overcomes this.
• Describe the replacement of the hard threshold with a smooth activation function, and explain why this is necessary for training with gradient descent.
• Trace a forward pass through a tiny, hand-specified neural network with one hidden layer.
• Explain precisely where this discipline's coverage of neural networks stops, and what is deliberately left to a dedicated deep learning discipline.

The previous concept's perceptron convergence theorem carried a real limitation, made explicit in its own worked example: a single perceptron can only ever represent a linear decision boundary, and simply cannot solve problems like the XOR pattern, regardless of how it is trained. This concept takes the one modification that fixes this — stacking many perceptron-like units into layers — and treats it as the deliberate bridge and stopping point between this discipline and ai-theory/deep-learning (an empty sibling discipline), rather than as a launch into the full mechanics of training deep networks.

Stacking units into layers

A single perceptron computes one linear score and applies a threshold. A neural network stacks many such units into layers: an input layer (the raw features), one or more hidden layers (each unit computing a weighted sum of the previous layer's outputs, then applying an activation function), and an output layer (producing the final prediction). Crucially, composing two purely linear layers back-to-back would collapse into a single linear function — mathematically identical to a single perceptron or logistic regression, buying nothing extra. The genuine expressive power of a multi-layer network comes specifically from inserting a nonlinear activation function between layers.

Replacing the hard threshold with a smooth activation

The perceptron's hard threshold (a step function) is not differentiable at zero and constant everywhere else — its derivative is zero almost everywhere, giving gradient descent nothing to work with. Neural networks replace this hard threshold with a smooth, differentiable activation function — historically the sigmoid already introduced for logistic regression, or (more common in modern practice) the ReLU function max(0, z) — specifically so that the network's overall output is a differentiable function of every weight in every layer, making it trainable end-to-end by gradient descent.

Why this is exactly where this discipline stops

Training a multi-layer network requires computing the gradient of the loss with respect to every weight in every layer — an algorithm called backpropagation, which applies the chain rule of calculus layer by layer. Backpropagation itself, along with the architectures (convolutional networks, recurrent networks, transformers) and the training tricks (dropout, batch normalization, modern optimizers beyond plain gradient descent) that make deep networks practical at scale, are deliberately left entirely to ai-theory/deep-learning. This discipline's role is narrower and specific: establish exactly why linear models hit a hard representational ceiling, and exactly what minimal change (layering plus nonlinearity) breaks through it, without developing the machinery to train networks with many layers efficiently.

Example 1: Solving XOR with one hidden layer

The XOR pattern that defeated a single perceptron in the previous concept — (1,1)→+1, (−1,−1)→+1, (1,−1)→−1, (−1,1)→−1 — can be solved exactly by a network with one hidden layer of just 2 units. Intuitively: one hidden unit can learn a boundary separating (1,1) from the rest, and a second hidden unit can learn a boundary separating (−1,−1) from the rest; the output layer then combines these two hidden signals (each individually still just a linear boundary) into a final decision that correctly reconstructs the XOR pattern no single linear boundary could represent. This is the concrete, checkable illustration of why layering with nonlinearity strictly increases expressive power beyond any single linear model.

Example 2: A hand-traced forward pass

Consider a tiny network: 2 inputs, 1 hidden layer with 2 ReLU units, 1 output unit. Given input x = (1, 2), hidden weights w₁ = (1, −1), w₂ = (0.5, 0.5) (with biases 0), and output weights w_out = (1, 1) (bias 0):

text
Hidden unit 1: z₁ = w₁·x = 1(1) + (−1)(2) = −1     →  ReLU(−1) = max(0, −1) = 0
Hidden unit 2: z₂ = w₂·x = 0.5(1) + 0.5(2) = 1.5   →  ReLU(1.5) = max(0, 1.5) = 1.5

Output: ŷ = w_out · (0, 1.5) = 1(0) + 1(1.5) = 1.5

Every step is a straightforward linear combination followed by a simple nonlinear function — the entire "forward pass" of a neural network is nothing more exotic than repeated applications of exactly this pattern, layer after layer.

• "A neural network with more layers is just a bigger linear model." Without a nonlinear activation function between layers, this would be exactly true — stacked linear layers collapse algebraically into one equivalent linear layer; the nonlinearity is precisely what prevents this collapse and is the entire source of a multi-layer network's extra expressive power.
• "Neural networks are trained the same way as the perceptron, just with more layers." The perceptron's mistake-driven update rule does not generalize to multi-layer networks — training requires backpropagation to compute how the loss depends on every weight in every layer, a substantially more involved algorithm this discipline deliberately does not develop, reserving it for ai-theory/deep-learning.
• "This discipline has now covered neural networks, so deep learning is redundant." This concept covers only the minimal representational idea — why stacking with nonlinearity breaks the linear ceiling — not the training algorithm, the architectures, or the scaling techniques that make deep networks practically useful on real, large-scale data; that is the entire, separate scope of the sibling discipline this concept exists to hand off to.

Stacking perceptron-like units into layers, with a smooth, differentiable activation function replacing the perceptron's hard threshold, breaks through the single linear boundary ceiling that limited every classifier so far in this discipline — concretely demonstrated by a 2-hidden-unit network solving the XOR problem a single perceptron provably cannot. This concept establishes exactly that representational leap and stops there, deliberately leaving backpropagation, deep architectures, and large-scale training techniques to the dedicated ai-theory/deep-learning discipline.