Computer Science › Intelligence + Theory
This discipline picks up exactly where `machine-learning` deliberately stopped — a network of layered perceptron-like units with nonlinear activations — and gives it real, precise machinery: every dense layer is literally a matrix multiply followed by a nonlinearity, and the entire forward pass is one composed function ready to be trained at scale.
The algorithm every network in this discipline is trained by: represent the forward pass as a computational graph of small, locally-differentiable operations, then walk that graph backward applying the multivariable chain rule one node at a time, so a single backward pass computes the gradient of the loss with respect to every weight in every layer at once.
The loss function nearly every classifier in this discipline is actually trained against: softmax turns a network's raw output scores into a probability distribution over classes, and cross-entropy — derived here directly from maximum likelihood on that distribution, the same MLE argument already used for logistic regression — measures exactly how far that distribution is from the true label.
Why a deep network cannot simply be initialized with all-zero or arbitrary random weights: backpropagation multiplies many layers' worth of derivatives together, so a systematic bias in the scale of those derivatives compounds multiplicatively with depth, shrinking gradients toward zero or blowing them up — and why calibrated schemes like Xavier and He initialization exist specifically to keep that product stable.
Plain gradient descent, already covered in full in `machine-learning`, is not re-derived here — it is extended: momentum accumulates a running velocity across steps to smooth out noisy gradients and speed through shallow regions, and Adam combines that momentum with a per-parameter adaptive learning rate, the default optimizer for nearly every architecture the rest of this discipline covers.
The same bias-variance tradeoff and overfitting already covered for linear models applies to deep networks with millions of parameters, only more urgently — weight decay is exactly the ridge penalty already covered, applied to a network's weights, while dropout is a technique with no shallow-model analogue at all: randomly deleting units during training to prevent any one of them from co-adapting into an overly specific, fragile feature detector.
A layer that does nothing but recenter and rescale its input to unit-Gaussian statistics using each mini-batch's own mean and variance — inserted between a linear layer and its activation specifically to keep the input distribution to every layer stable as the weights below it change during training, letting much larger learning rates converge reliably.
The operation this discipline's anchor course (CS231n) exists to teach in depth: a small learned filter slides across an image computing a dot product — the exact operation already covered as a scalar vector-algebra fact — at every position, producing a feature map that lights up wherever the pattern the filter learned to detect appears.
The mechanics that turn a single convolution into a practical, trainable layer — stride controls how far the filter jumps between positions, padding controls what happens at the border, pooling discards spatial detail on purpose to build in some translation tolerance — and the deeper reason a convolutional layer needs orders of magnitude fewer parameters than a fully-connected one: every position on the image reuses the exact same filter weights.
How stacking convolution/pooling blocks evolved from LeNet to AlexNet, VGG, and ResNet — and why simply stacking more layers eventually made accuracy worse until residual (skip) connections let a layer learn a correction to its input rather than a whole new representation, directly fighting the vanishing-gradient problem for very deep networks — plus why this entire discipline's core operation, the matrix multiply, is exactly the kind of massively data-parallel workload GPUs were built for.
The architecture built for data with an inherent order — text, audio, time series — where a single hidden state is carried forward and updated at every timestep using the same shared weights, and training unrolls that recurrence into one long computational graph so backpropagation (already covered in full) can run across time as well as across layers.
The same vanishing-gradient problem already diagnosed for very deep feedforward networks reappears in an even sharper form once a recurrent network is unrolled across a long sequence — and the gating mechanism that fixes it: LSTM's forget, input, and output gates (with GRU as a simplified variant) maintain a memory cell with a near-constant error-carrying path across time, letting gradients survive dozens or hundreds of timesteps instead of vanishing.
A different answer to the same problem LSTM/GRU gating manages only partially — letting every output position look directly at every input position, rather than compressing the whole sequence's history through one hidden state — computed with nothing more exotic than the dot product: a query vector is compared against every key vector, and the resulting scores weight a sum of value vectors.
Applying attention to a sequence's own tokens, all at once, removes the strict left-to-right dependency that forces a recurrent network to process one timestep at a time — every position's attention can be computed in parallel — but that same parallelism throws away word order entirely unless it is reinjected explicitly, which is exactly what positional encoding does.
Assembling self-attention, multi-head attention, residual connections (the same skip-connection idea already covered for very deep CNNs), and per-position feed-forward layers into the architecture that now dominates language, vision, and multimodal models alike — the model this entire attention cluster was building toward.
The unifying idea underneath every architecture this discipline has covered: a CNN's early layers learn generic edge and color detectors while its later layers learn increasingly task-specific features, and a transformer's embeddings play exactly the same role for tokens — in both cases, the network is not being handed features by a human, it is learning its own representation of the input directly from data, which is precisely why a model trained on one large dataset transfers so well to a new, related task.
The first generative model in this discipline: an encoder compresses an input down to a small bottleneck vector and a decoder reconstructs the original from just that bottleneck, trained end-to-end to minimize reconstruction error — a direct, nonlinear generalization of PCA (already covered as the linear case of exactly this same compression idea), and the foundation the more elaborate generative architectures that follow are built on.
A deliberately survey-level look at the two generative families that most visibly define today's AI applications — a generative adversarial network's generator and discriminator locked in a minimax game, and a diffusion model's iterative denoising process — covering the core idea and why each moved past a plain autoencoder's blurrier reconstructions, while explicitly leaving the full training dynamics and mathematical derivations to a future, more advanced discipline.
A capstone walkthrough comparing every architecture family covered in this discipline — plain multilayer networks, CNNs, RNN/LSTM/GRU, and Transformers — by the structure of the data they assume (grid, sequence, set) and the training concerns (initialization, regularization, optimizer, compute) that apply to all of them, extending the analogous decision guide that closed `machine-learning` from classical models into the deep architectures this discipline added.
Part of the Computer Science curriculum.
This discipline picks up exactly where `machine-learning` deliberately stopped — a network of layered perceptron-like units with nonlinear activations — and gives it real, precise machinery: every dense layer is literally a matrix multiply followed by a nonlinearity, and the entire forward pass is one composed function ready to be trained at scale.
The algorithm every network in this discipline is trained by: represent the forward pass as a computational graph of small, locally-differentiable operations, then walk that graph backward applying the multivariable chain rule one node at a time, so a single backward pass computes the gradient of the loss with respect to every weight in every layer at once.
The loss function nearly every classifier in this discipline is actually trained against: softmax turns a network's raw output scores into a probability distribution over classes, and cross-entropy — derived here directly from maximum likelihood on that distribution, the same MLE argument already used for logistic regression — measures exactly how far that distribution is from the true label.
Why a deep network cannot simply be initialized with all-zero or arbitrary random weights: backpropagation multiplies many layers' worth of derivatives together, so a systematic bias in the scale of those derivatives compounds multiplicatively with depth, shrinking gradients toward zero or blowing them up — and why calibrated schemes like Xavier and He initialization exist specifically to keep that product stable.
Plain gradient descent, already covered in full in `machine-learning`, is not re-derived here — it is extended: momentum accumulates a running velocity across steps to smooth out noisy gradients and speed through shallow regions, and Adam combines that momentum with a per-parameter adaptive learning rate, the default optimizer for nearly every architecture the rest of this discipline covers.
The same bias-variance tradeoff and overfitting already covered for linear models applies to deep networks with millions of parameters, only more urgently — weight decay is exactly the ridge penalty already covered, applied to a network's weights, while dropout is a technique with no shallow-model analogue at all: randomly deleting units during training to prevent any one of them from co-adapting into an overly specific, fragile feature detector.
A layer that does nothing but recenter and rescale its input to unit-Gaussian statistics using each mini-batch's own mean and variance — inserted between a linear layer and its activation specifically to keep the input distribution to every layer stable as the weights below it change during training, letting much larger learning rates converge reliably.
The operation this discipline's anchor course (CS231n) exists to teach in depth: a small learned filter slides across an image computing a dot product — the exact operation already covered as a scalar vector-algebra fact — at every position, producing a feature map that lights up wherever the pattern the filter learned to detect appears.
The mechanics that turn a single convolution into a practical, trainable layer — stride controls how far the filter jumps between positions, padding controls what happens at the border, pooling discards spatial detail on purpose to build in some translation tolerance — and the deeper reason a convolutional layer needs orders of magnitude fewer parameters than a fully-connected one: every position on the image reuses the exact same filter weights.
How stacking convolution/pooling blocks evolved from LeNet to AlexNet, VGG, and ResNet — and why simply stacking more layers eventually made accuracy worse until residual (skip) connections let a layer learn a correction to its input rather than a whole new representation, directly fighting the vanishing-gradient problem for very deep networks — plus why this entire discipline's core operation, the matrix multiply, is exactly the kind of massively data-parallel workload GPUs were built for.
The architecture built for data with an inherent order — text, audio, time series — where a single hidden state is carried forward and updated at every timestep using the same shared weights, and training unrolls that recurrence into one long computational graph so backpropagation (already covered in full) can run across time as well as across layers.
The same vanishing-gradient problem already diagnosed for very deep feedforward networks reappears in an even sharper form once a recurrent network is unrolled across a long sequence — and the gating mechanism that fixes it: LSTM's forget, input, and output gates (with GRU as a simplified variant) maintain a memory cell with a near-constant error-carrying path across time, letting gradients survive dozens or hundreds of timesteps instead of vanishing.
A different answer to the same problem LSTM/GRU gating manages only partially — letting every output position look directly at every input position, rather than compressing the whole sequence's history through one hidden state — computed with nothing more exotic than the dot product: a query vector is compared against every key vector, and the resulting scores weight a sum of value vectors.
Applying attention to a sequence's own tokens, all at once, removes the strict left-to-right dependency that forces a recurrent network to process one timestep at a time — every position's attention can be computed in parallel — but that same parallelism throws away word order entirely unless it is reinjected explicitly, which is exactly what positional encoding does.
Assembling self-attention, multi-head attention, residual connections (the same skip-connection idea already covered for very deep CNNs), and per-position feed-forward layers into the architecture that now dominates language, vision, and multimodal models alike — the model this entire attention cluster was building toward.
The unifying idea underneath every architecture this discipline has covered: a CNN's early layers learn generic edge and color detectors while its later layers learn increasingly task-specific features, and a transformer's embeddings play exactly the same role for tokens — in both cases, the network is not being handed features by a human, it is learning its own representation of the input directly from data, which is precisely why a model trained on one large dataset transfers so well to a new, related task.
The first generative model in this discipline: an encoder compresses an input down to a small bottleneck vector and a decoder reconstructs the original from just that bottleneck, trained end-to-end to minimize reconstruction error — a direct, nonlinear generalization of PCA (already covered as the linear case of exactly this same compression idea), and the foundation the more elaborate generative architectures that follow are built on.
A deliberately survey-level look at the two generative families that most visibly define today's AI applications — a generative adversarial network's generator and discriminator locked in a minimax game, and a diffusion model's iterative denoising process — covering the core idea and why each moved past a plain autoencoder's blurrier reconstructions, while explicitly leaving the full training dynamics and mathematical derivations to a future, more advanced discipline.
A capstone walkthrough comparing every architecture family covered in this discipline — plain multilayer networks, CNNs, RNN/LSTM/GRU, and Transformers — by the structure of the data they assume (grid, sequence, set) and the training concerns (initialization, regularization, optimizer, compute) that apply to all of them, extending the analogous decision guide that closed `machine-learning` from classical models into the deep architectures this discipline added.