Computer Science › Intelligence + Theory
The three-way split that organizes the entire field of machine learning — supervised (learning from labeled examples), unsupervised (finding structure with no labels at all), and reinforcement (learning from reward through interaction) — and why this discipline covers the first two in depth while pointing to material already published for the third.
Before writing a single learning algorithm, the honest question this discipline opens with: can a rule learned from a finite sample of data ever be trusted to work on data it has never seen? The Hoeffding bound gives a real, quantitative answer.
The practical discipline that turns the generalization question into something you can actually measure: holding out data the model never trains on, and never touching the true test set until every other decision is already made.
The single most-used model in all of statistics and machine learning, arriving here as no new mathematics at all — it is exactly the normal-equation problem already solved geometrically and via optimization in this curriculum's linear algebra discipline, now reframed as prediction on unseen data.
Why minimizing squared error is not an arbitrary choice: assuming the noise around the true line is normally distributed, maximizing the likelihood of the observed data produces exactly the same least-squares solution already derived geometrically.
Turning the informal gradient already introduced in this curriculum's linear algebra discipline into a full iterative algorithm — repeatedly stepping opposite the gradient until a loss function's minimum is reached, the one optimization technique nearly every model in this discipline ultimately reduces to.
Adapting the linear-model-plus-gradient-descent machinery just built to a fundamentally different task — predicting a category, not a number — by squashing a linear score through a sigmoid and drawing the boundary where it crosses one-half.
A second, genuinely different way to classify: instead of directly modeling the decision boundary (as logistic regression does), model how each class generates its data and let Bayes' theorem — already covered in full with a real diagnostic-test example — turn that into a prediction.
Decomposing a model's prediction error into two competing sources — bias, from a model too simple to capture the true pattern, and variance, from a model so flexible it fits the noise — with a real worked numeric decomposition, not just the qualitative story.
A precise measure of how much a model class can possibly overfit, before it ever sees any data — the VC dimension counts the largest set of points a model can shatter, and directly predicts how much data it takes to learn reliably.
Fighting overfitting directly by penalizing large weights in the loss function — ridge regression's smooth circular penalty shrinks every weight a little, while lasso's diamond-shaped penalty can shrink some weights to exactly zero, a real geometric difference with real consequences.
Squeezing a reliable estimate of a model's true generalization error out of a single, finite dataset — k-fold cross-validation trains and validates on every possible split, then averages, so no single lucky or unlucky split decides which model wins.
Why raw accuracy lies on imbalanced data, and the confusion-matrix-derived metrics — precision, recall, F1, and the ROC curve's true-positive-vs-false-positive tradeoff — that a real classifier must actually be judged on.
A classifier built from nothing more exotic than the binary tree already covered as a data structure — here, each split is chosen by the feature and threshold that most reduces impurity, producing a model a human can read straight off the page.
Combining many individually weak or overfit trees into one strong model — bagging averages many independent, high-variance trees to cancel out their noise, while boosting builds trees in sequence, each one correcting the last one's mistakes.
Choosing not just any separating line between two classes, but the one with the widest possible margin on both sides — the handful of points that margin actually touches, the support vectors, are the only data the final classifier depends on.
Making a linear classifier separate data that is not linearly separable at all, by replacing every dot product — the one operation SVMs actually use — with a kernel function that computes the same value as if the data had been mapped into a much higher-dimensional space, without ever building that space.
The first unsupervised algorithm in this discipline: with no labels at all, alternately assign each point to its nearest of k cluster centers and recompute each center as the mean of its assigned points, until nothing moves.
A softer, probabilistic generalization of k-means — every point gets a probability of belonging to each of several overlapping normal distributions, fit by alternating between guessing those probabilities (E-step) and re-fitting the distributions to match (M-step).
Compressing high-dimensional data down to its most informative few directions — literally the eigenvectors of the data's own covariance matrix, the exact guarantee about symmetric matrices already proven in this curriculum's linear algebra discipline, now put to direct use.
The oldest learning algorithm in this cluster and the direct ancestor of every neural network — a single linear threshold unit that adjusts its own weights, one misclassified example at a time, until (if the data is linearly separable) it converges exactly.
Stacking perceptron-like units into layers and replacing the hard threshold with a smooth activation function — enough to escape the single perceptron's fundamental limit of only ever drawing a straight line, and the exact point where this discipline deliberately stops, handing off to a dedicated deep learning discipline.
A capstone walkthrough comparing every model family covered in this discipline — regression, generative and discriminative classifiers, trees and ensembles, SVMs, clustering, and neural networks — by interpretability, data size, and nonlinearity, and validating the final choice end to end.
Part of the Computer Science curriculum.
The three-way split that organizes the entire field of machine learning — supervised (learning from labeled examples), unsupervised (finding structure with no labels at all), and reinforcement (learning from reward through interaction) — and why this discipline covers the first two in depth while pointing to material already published for the third.
Before writing a single learning algorithm, the honest question this discipline opens with: can a rule learned from a finite sample of data ever be trusted to work on data it has never seen? The Hoeffding bound gives a real, quantitative answer.
The practical discipline that turns the generalization question into something you can actually measure: holding out data the model never trains on, and never touching the true test set until every other decision is already made.
The single most-used model in all of statistics and machine learning, arriving here as no new mathematics at all — it is exactly the normal-equation problem already solved geometrically and via optimization in this curriculum's linear algebra discipline, now reframed as prediction on unseen data.
Why minimizing squared error is not an arbitrary choice: assuming the noise around the true line is normally distributed, maximizing the likelihood of the observed data produces exactly the same least-squares solution already derived geometrically.
Turning the informal gradient already introduced in this curriculum's linear algebra discipline into a full iterative algorithm — repeatedly stepping opposite the gradient until a loss function's minimum is reached, the one optimization technique nearly every model in this discipline ultimately reduces to.
Adapting the linear-model-plus-gradient-descent machinery just built to a fundamentally different task — predicting a category, not a number — by squashing a linear score through a sigmoid and drawing the boundary where it crosses one-half.
A second, genuinely different way to classify: instead of directly modeling the decision boundary (as logistic regression does), model how each class generates its data and let Bayes' theorem — already covered in full with a real diagnostic-test example — turn that into a prediction.
Decomposing a model's prediction error into two competing sources — bias, from a model too simple to capture the true pattern, and variance, from a model so flexible it fits the noise — with a real worked numeric decomposition, not just the qualitative story.
A precise measure of how much a model class can possibly overfit, before it ever sees any data — the VC dimension counts the largest set of points a model can shatter, and directly predicts how much data it takes to learn reliably.
Fighting overfitting directly by penalizing large weights in the loss function — ridge regression's smooth circular penalty shrinks every weight a little, while lasso's diamond-shaped penalty can shrink some weights to exactly zero, a real geometric difference with real consequences.
Squeezing a reliable estimate of a model's true generalization error out of a single, finite dataset — k-fold cross-validation trains and validates on every possible split, then averages, so no single lucky or unlucky split decides which model wins.
Why raw accuracy lies on imbalanced data, and the confusion-matrix-derived metrics — precision, recall, F1, and the ROC curve's true-positive-vs-false-positive tradeoff — that a real classifier must actually be judged on.
A classifier built from nothing more exotic than the binary tree already covered as a data structure — here, each split is chosen by the feature and threshold that most reduces impurity, producing a model a human can read straight off the page.
Combining many individually weak or overfit trees into one strong model — bagging averages many independent, high-variance trees to cancel out their noise, while boosting builds trees in sequence, each one correcting the last one's mistakes.
Choosing not just any separating line between two classes, but the one with the widest possible margin on both sides — the handful of points that margin actually touches, the support vectors, are the only data the final classifier depends on.
Making a linear classifier separate data that is not linearly separable at all, by replacing every dot product — the one operation SVMs actually use — with a kernel function that computes the same value as if the data had been mapped into a much higher-dimensional space, without ever building that space.
The first unsupervised algorithm in this discipline: with no labels at all, alternately assign each point to its nearest of k cluster centers and recompute each center as the mean of its assigned points, until nothing moves.
A softer, probabilistic generalization of k-means — every point gets a probability of belonging to each of several overlapping normal distributions, fit by alternating between guessing those probabilities (E-step) and re-fitting the distributions to match (M-step).
Compressing high-dimensional data down to its most informative few directions — literally the eigenvectors of the data's own covariance matrix, the exact guarantee about symmetric matrices already proven in this curriculum's linear algebra discipline, now put to direct use.
The oldest learning algorithm in this cluster and the direct ancestor of every neural network — a single linear threshold unit that adjusts its own weights, one misclassified example at a time, until (if the data is linearly separable) it converges exactly.
Stacking perceptron-like units into layers and replacing the hard threshold with a smooth activation function — enough to escape the single perceptron's fundamental limit of only ever drawing a straight line, and the exact point where this discipline deliberately stops, handing off to a dedicated deep learning discipline.
A capstone walkthrough comparing every model family covered in this discipline — regression, generative and discriminative classifiers, trees and ensembles, SVMs, clustering, and neural networks — by interpretability, data size, and nonlinearity, and validating the final choice end to end.