Computer Science › Intelligence + Theory
The KDD process from raw data to validated knowledge, and the task taxonomy of classification, regression, clustering, association, and anomaly detection, with where this curriculum covers each one.
Cleaning data before mining: missing-value mechanisms and imputation, min-max and z-score scaling, discretization, and the leakage caused by fitting a preprocessing step on the whole dataset.
Warehouses versus operational databases, star and snowflake schemas, data marts, and the OLAP operations roll-up, drill-down, slice, dice, and pivot, with a book-versus-today note on columnar cloud warehouses.
Distance and similarity for numeric, binary, and text data: Euclidean, Manhattan, Minkowski, Jaccard, Hamming, and cosine, and why the choice and the scaling change the result.
k-NN as lazy learning: classification and regression by neighbors, choosing k, the curse of dimensionality, and the index structures that make queries practical.
Itemsets and support, the anti-monotone property that prunes the search, and Apriori's level-wise candidate generation worked on a small transaction set.
Mining frequent itemsets without candidate generation: building the FP-tree, mining conditional pattern bases, and when this beats Apriori.
From frequent itemsets to rules: support, confidence, and lift, and why high confidence alone can mislead.
Agglomerative and divisive clustering, single, complete, average, and Ward linkage, reading a dendrogram, and how this differs from k-means.
DBSCAN's core, border, and noise points, density reachability, choosing eps and minPts, and clusters of arbitrary shape.
Internal and external validation with silhouette, Davies-Bouldin, and the adjusted Rand index, choosing the number of clusters, and when these metrics mislead.
Outliers by the interquartile range, by distance, and by local density with LOF, and how to decide whether an outlier is an error or the finding.
Turning text into vectors: tokenization, stop words, stemming, bag of words, TF-IDF weighting, and cosine similarity between documents.
A worked pipeline from raw documents through TF-IDF, dimensionality reduction, clustering, and evaluation, ending with reading the top terms of each cluster.
What changes when patterns and clusters no longer fit in memory: MapReduce rounds, Spark's in-memory model, and parallel frequent-pattern mining and k-means. Mahout appears only as a book-versus-today note.
A decision guide for the whole discipline: given the data and the question, choose a mining task and a technique, and name what would invalidate the result.
Part of the Computer Science curriculum.
The KDD process from raw data to validated knowledge, and the task taxonomy of classification, regression, clustering, association, and anomaly detection, with where this curriculum covers each one.
Cleaning data before mining: missing-value mechanisms and imputation, min-max and z-score scaling, discretization, and the leakage caused by fitting a preprocessing step on the whole dataset.
Warehouses versus operational databases, star and snowflake schemas, data marts, and the OLAP operations roll-up, drill-down, slice, dice, and pivot, with a book-versus-today note on columnar cloud warehouses.
Distance and similarity for numeric, binary, and text data: Euclidean, Manhattan, Minkowski, Jaccard, Hamming, and cosine, and why the choice and the scaling change the result.
k-NN as lazy learning: classification and regression by neighbors, choosing k, the curse of dimensionality, and the index structures that make queries practical.
Itemsets and support, the anti-monotone property that prunes the search, and Apriori's level-wise candidate generation worked on a small transaction set.
Mining frequent itemsets without candidate generation: building the FP-tree, mining conditional pattern bases, and when this beats Apriori.
From frequent itemsets to rules: support, confidence, and lift, and why high confidence alone can mislead.
Agglomerative and divisive clustering, single, complete, average, and Ward linkage, reading a dendrogram, and how this differs from k-means.
DBSCAN's core, border, and noise points, density reachability, choosing eps and minPts, and clusters of arbitrary shape.
Internal and external validation with silhouette, Davies-Bouldin, and the adjusted Rand index, choosing the number of clusters, and when these metrics mislead.
Outliers by the interquartile range, by distance, and by local density with LOF, and how to decide whether an outlier is an error or the finding.
Turning text into vectors: tokenization, stop words, stemming, bag of words, TF-IDF weighting, and cosine similarity between documents.
A worked pipeline from raw documents through TF-IDF, dimensionality reduction, clustering, and evaluation, ending with reading the top terms of each cluster.
What changes when patterns and clusters no longer fit in memory: MapReduce rounds, Spark's in-memory model, and parallel frequent-pattern mining and k-means. Mahout appears only as a book-versus-today note.
A decision guide for the whole discipline: given the data and the question, choose a mining task and a technique, and name what would invalidate the result.