Computer Science › Intelligence + Theory
Unlike Labs 1 through 3, these two algorithms have no labels to fit against, no loss function measuring distance from a known right answer, which is the concrete, hands-on lesson this lab is built to deliver: k-Means Clustering's assign-then-update loop implemented from scratch and run until convergence, and Principal Component Analysis's own eigendecomposition of a real dataset's covariance matrix, verified not against a label but against a directly checkable geometric property, k-means's total within-cluster distance strictly decreasing every iteration, and PCA's own top components capturing the claimed, computed fraction of total variance.
Linear Regression as Least Squares and Gradient Descent as a General Optimizer already prove, on paper, that minimizing mean squared error has a closed-form solution and that gradient descent converges toward it iteratively; this lab implements both, in plain NumPy with no scikit-learn anywhere in the loop, on a real, small dataset, and verifies the two independently-computed answers, closed-form and iterative, agree with each other, which is the whole real point: proof of convergence and actually observed convergence are two different things, and this lab is where the second one gets checked.
This lab opens the second arc of this discipline, applying `graduate-studies/research-statistics`'s own hypothesis-and-evidence framework to a genuinely hands-on ML setting, not just a real dataset, but a real, still-open question about it, sharpened, exactly as Hypotheses, Questions, and Forms of Evidence describes, into a specific, falsifiable claim (a particular feature's coefficient sign, a specific accuracy threshold Lab 1's model should clear) before Lab 6's dataset work or Lab 7's model selection even begins, so every later lab in this arc is answering a question stated precisely enough in advance to actually be wrong.
Training, Test, and Validation Splits already establishes why a model can only be honestly evaluated on data it never touched during training; this lab makes that discipline concrete and, deliberately, makes its violation concrete too: a correct split implemented first, then a second, intentionally broken version that leaks test-set information into training (through a preprocessing step, like a feature-scaling normalization, fit on the full dataset before splitting), measuring directly how much that leak inflates the reported accuracy, exactly the honest-versus-dishonest evaluation gap Cross-Validation's own k-fold discipline exists to close.
This lab reuses Lab 1's own gradient-descent loop unchanged, swapping only the loss function, cross-entropy in place of mean squared error, and the output nonlinearity, sigmoid in place of the identity, and verifies the resulting classifier's decision boundary against Logistic Regression and Decision Boundaries' own account of where that boundary sits (the set of points where the sigmoid's output crosses 0.5), plotted directly against a real, small, linearly separable dataset rather than only asserted algebraically.
Lab 2's logistic regression is, in the exact sense Multilayer Networks as Composed Linear Transformations describes, a neural network with zero hidden layers; this lab adds one hidden layer and implements backpropagation's own chain-rule gradient computation by hand, layer by layer, then verifies every hand-derived gradient against numerical gradient checking (a small finite-difference perturbation of each weight), the standard, real way Backpropagation: The Chain Rule Through a Computational Graph's own correctness is actually confirmed in practice, not merely trusted from the derivation.
The Bias-Variance Tradeoff and Overfitting and the VC Dimension already predict, theoretically, a U-shaped validation-error curve as model complexity increases, underfitting on one side, overfitting on the other; this lab sweeps model complexity directly, polynomial degree for Lab 1's regressor, hidden-layer width for Lab 3's small network, plots training error against held-out validation error across that sweep, and measures the exact point the curve turns, then applies Regularization: Ridge and Lasso's own penalty term and confirms, empirically, that it shifts that turning point rather than merely being asserted to.
This lab applies `graduate-studies/research-statistics`'s own baseline-and-robustness discipline directly to the models this arc has built: Lab 7's best-selected model run against a real, fair baseline (an untuned default configuration, not a strawman), evaluated with Evaluation Metrics: Precision, Recall, and ROC's own full metric set rather than accuracy alone, across multiple random seeds to report honest variability, and written up as a short, real report answering Lab 5's original hypothesis directly, supported or refuted, with the actual evidence, not just a final number.
The NeurIPS Reproducibility Program's own real, peer-reviewed report on machine learning reproducibility documents, with real, collected data from real submitted papers, how often published ML results fail to reproduce and what specifically tends to be missing; this lab applies that report's own checklist, exact hyperparameters, exact data preprocessing steps, exact random seeds, hardware and software versions, directly to Lab 8's own experiment, first hiding the checklist items deliberately (matching a typically under-documented paper) to show a fresh implementation attempt fails to reproduce the reported number, then filling in every checklist item and confirming that it now does.
This capstone runs the full cycle this discipline's own topic sequence names, question, hypothesis, dataset, model, experiment, evaluation, reproduce, challenge, end to end and under real, Kaggle-style competitive pressure: a dataset whose true test-set labels are genuinely withheld, a fixed deadline, and a single held-out leaderboard score as the only feedback, forcing every earlier lab's discipline, honest splits from Lab 6, principled model selection from Lab 7, a real baseline from Lab 8, into one real decision under uncertainty, closing exactly where Choosing and Validating a Model: A Decision Guide's own theoretical framework opened, but now applied under a real, binding deadline instead of worked through at leisure.
Part of the Computer Science curriculum.
Unlike Labs 1 through 3, these two algorithms have no labels to fit against, no loss function measuring distance from a known right answer, which is the concrete, hands-on lesson this lab is built to deliver: k-Means Clustering's assign-then-update loop implemented from scratch and run until convergence, and Principal Component Analysis's own eigendecomposition of a real dataset's covariance matrix, verified not against a label but against a directly checkable geometric property, k-means's total within-cluster distance strictly decreasing every iteration, and PCA's own top components capturing the claimed, computed fraction of total variance.
Linear Regression as Least Squares and Gradient Descent as a General Optimizer already prove, on paper, that minimizing mean squared error has a closed-form solution and that gradient descent converges toward it iteratively; this lab implements both, in plain NumPy with no scikit-learn anywhere in the loop, on a real, small dataset, and verifies the two independently-computed answers, closed-form and iterative, agree with each other, which is the whole real point: proof of convergence and actually observed convergence are two different things, and this lab is where the second one gets checked.
This lab opens the second arc of this discipline, applying `graduate-studies/research-statistics`'s own hypothesis-and-evidence framework to a genuinely hands-on ML setting, not just a real dataset, but a real, still-open question about it, sharpened, exactly as Hypotheses, Questions, and Forms of Evidence describes, into a specific, falsifiable claim (a particular feature's coefficient sign, a specific accuracy threshold Lab 1's model should clear) before Lab 6's dataset work or Lab 7's model selection even begins, so every later lab in this arc is answering a question stated precisely enough in advance to actually be wrong.
Training, Test, and Validation Splits already establishes why a model can only be honestly evaluated on data it never touched during training; this lab makes that discipline concrete and, deliberately, makes its violation concrete too: a correct split implemented first, then a second, intentionally broken version that leaks test-set information into training (through a preprocessing step, like a feature-scaling normalization, fit on the full dataset before splitting), measuring directly how much that leak inflates the reported accuracy, exactly the honest-versus-dishonest evaluation gap Cross-Validation's own k-fold discipline exists to close.
This lab reuses Lab 1's own gradient-descent loop unchanged, swapping only the loss function, cross-entropy in place of mean squared error, and the output nonlinearity, sigmoid in place of the identity, and verifies the resulting classifier's decision boundary against Logistic Regression and Decision Boundaries' own account of where that boundary sits (the set of points where the sigmoid's output crosses 0.5), plotted directly against a real, small, linearly separable dataset rather than only asserted algebraically.
Lab 2's logistic regression is, in the exact sense Multilayer Networks as Composed Linear Transformations describes, a neural network with zero hidden layers; this lab adds one hidden layer and implements backpropagation's own chain-rule gradient computation by hand, layer by layer, then verifies every hand-derived gradient against numerical gradient checking (a small finite-difference perturbation of each weight), the standard, real way Backpropagation: The Chain Rule Through a Computational Graph's own correctness is actually confirmed in practice, not merely trusted from the derivation.
The Bias-Variance Tradeoff and Overfitting and the VC Dimension already predict, theoretically, a U-shaped validation-error curve as model complexity increases, underfitting on one side, overfitting on the other; this lab sweeps model complexity directly, polynomial degree for Lab 1's regressor, hidden-layer width for Lab 3's small network, plots training error against held-out validation error across that sweep, and measures the exact point the curve turns, then applies Regularization: Ridge and Lasso's own penalty term and confirms, empirically, that it shifts that turning point rather than merely being asserted to.
This lab applies `graduate-studies/research-statistics`'s own baseline-and-robustness discipline directly to the models this arc has built: Lab 7's best-selected model run against a real, fair baseline (an untuned default configuration, not a strawman), evaluated with Evaluation Metrics: Precision, Recall, and ROC's own full metric set rather than accuracy alone, across multiple random seeds to report honest variability, and written up as a short, real report answering Lab 5's original hypothesis directly, supported or refuted, with the actual evidence, not just a final number.
The NeurIPS Reproducibility Program's own real, peer-reviewed report on machine learning reproducibility documents, with real, collected data from real submitted papers, how often published ML results fail to reproduce and what specifically tends to be missing; this lab applies that report's own checklist, exact hyperparameters, exact data preprocessing steps, exact random seeds, hardware and software versions, directly to Lab 8's own experiment, first hiding the checklist items deliberately (matching a typically under-documented paper) to show a fresh implementation attempt fails to reproduce the reported number, then filling in every checklist item and confirming that it now does.
This capstone runs the full cycle this discipline's own topic sequence names, question, hypothesis, dataset, model, experiment, evaluation, reproduce, challenge, end to end and under real, Kaggle-style competitive pressure: a dataset whose true test-set labels are genuinely withheld, a fixed deadline, and a single held-out leaderboard score as the only feedback, forcing every earlier lab's discipline, honest splits from Lab 6, principled model selection from Lab 7, a real baseline from Lab 8, into one real decision under uncertainty, closing exactly where Choosing and Validating a Model: A Decision Guide's own theoretical framework opened, but now applied under a real, binding deadline instead of worked through at leisure.