Representation Learning: The Network Learns Its Own Features

• State the unifying idea of representation learning: a network's internal layers learn a transformed encoding of the input, not just intermediate arithmetic on the way to a final answer.
• Explain the generic-to-specific progression of learned features across a CNN's layers, and why this makes transfer learning possible.
• Explain how a Transformer's token embeddings play the analogous role for sequences that CNN feature maps play for images.
• Distinguish representation learning from the classical feature engineering approach ai-theory/machine-learning largely assumed.

Every classical model in ai-theory/machine-learning — linear regression, logistic regression, SVMs, decision trees — operates on a fixed, human-chosen set of input features; the burden of deciding what to measure about the raw data (which pixel statistics, which text features, which engineered ratios) falls on whoever prepared the dataset before the model ever sees it. Every architecture covered in this discipline so far — CNNs' hierarchy of convolutional filters, Transformers' token embeddings and attention-derived representations — does something categorically different: the network's own internal layers learn what features are useful, directly from data, as an integral part of training via backpropagation. This concept names and unifies that idea explicitly, tying together threads that appeared separately in the CNN and Transformer clusters without yet being stated as one general principle.

From hand-engineered features to learned ones

ai-theory/machine-learning's principal-component-analysis was actually a partial exception within that discipline — PCA learns its principal directions from the data's own covariance structure, rather than having them handed to it — but it is still a fixed, one-shot linear transformation, computed once before any predictive model is trained on top of it. Deep networks generalize this idea much further: every layer's transformation is learned end-to-end, jointly with the final task the network is trained to solve, and the transformation can be arbitrarily nonlinear rather than the single linear projection PCA computes. This is the precise sense in which representation learning is a genuine expansion of the same underlying idea PCA already introduced, not an unrelated new concept.

The generic-to-specific progression in CNNs

cnn-architectures-and-residual-connections' transfer-learning material already contains the concrete evidence for this concept's central claim: a trained CNN's earliest convolutional layers learn generic, broadly-reusable detectors (edges, color blobs, simple textures) that have nothing to do with the specific classes the network was trained to distinguish, while its later layers combine those generic detectors into increasingly complex, increasingly task-specific patterns (an eye, a wheel, a particular texture combination unique to one class). This progression — generic and broadly transferable in early layers, specific and task-tuned in later ones — is exactly why transfer learning works at all: reusing a network's early layers on a new, related task reuses feature detectors that were never really specific to the original task in the first place.

Embeddings: the same idea for tokens and sequences

A Transformer's input embedding — the vector each input token is converted into before any self-attention is applied — plays the direct sequence-modeling analogue of a CNN's early feature maps: a learned representation of each token in a continuous space where, after training, semantically or functionally related tokens tend to end up nearby. Self-attention then builds increasingly contextualized representations of each position — the same generic-to-contextual progression already seen for CNNs, just built from attention-weighted combinations of other positions' representations rather than local convolutional filters. In both architectures, a raw input (a pixel grid, a sequence of discrete tokens) is progressively transformed, layer by layer, into representations that make the network's final task easier to solve — this is representation learning, viewed as one idea running through every architecture this discipline has covered.

Example 1: What transfer learning reuses, concretely

Consider a CNN trained on a large, general-purpose image dataset to classify 1,000 everyday object categories. Its early convolutional layers, by the generic-to-specific progression above, have learned filters detecting edges, corners, and simple color transitions — patterns that appear in essentially every natural image, regardless of what the image ultimately depicts. Reusing exactly those early layers as a fixed feature extractor for a new task — say, classifying microscope images of cells, a domain the network never saw during training — works precisely because those early filters were never actually specialized to the original 1,000 categories; they encode broadly useful visual structure, which is the concrete content behind the general claim "the network learns its own features."

Example 2: Embedding space as a geometric object

Suppose a Transformer's learned token embeddings place the tokens "king," "queen," "man," and "woman" such that the vector difference embedding("king") − embedding("man") is approximately equal to embedding("queen") − embedding("woman") — both differences point in roughly the same direction in the embedding space, corresponding to whatever relationship the network learned separates the "royal" pair from the "non-royal" pair. Nothing about the network's architecture explicitly programmed this geometric structure to exist; it is a byproduct of training the embeddings, jointly with every other weight, to make the network's actual task (predicting text, or whatever downstream objective it was trained on) as accurate as possible — the geometry emerges from the learning process, exactly as the CNN's edge detectors emerged from image classification training, not from any hand-specified rule about how word relationships should be arranged in space.

• "Representation learning is a technique specific to CNNs or specific to Transformers." It is the general property that any deep network's internal layers learn a transformed encoding of the input as an integral, end-to-end-trained part of solving its task — this concept deliberately draws the parallel across CNNs and Transformers precisely because the same underlying idea manifests differently depending on the architecture's specific structure (local filters for images, attention-derived contextual vectors for sequences).
• "A pretrained model's early layers are already perfectly general and never need adjusting for a new task." Transfer learning's guidelines (already covered) distinguish "similar to the original data" from "very different from the original data" precisely because generic features transfer better the more similar the new task's data is to what the original features were learned from — genericity is a matter of degree, not an absolute guarantee.
• "Feature engineering (the classical ai-theory/machine-learning approach) is now obsolete." For problems with modest data, well-understood domain structure, or a need for interpretability, hand-engineered features and classical models remain a real, reasonable choice — representation learning trades the effort of hand-designing features for the requirement of enough data and compute to learn good ones automatically, which is a genuine tradeoff, not a strict improvement in every situation.

Every classical model ai-theory/machine-learning covered operated on fixed, human-chosen features; every deep architecture this discipline has covered instead learns its own internal representation of the input end-to-end, jointly with the task it is trained on — a direct, much more general extension of the same idea PCA (a fixed, one-shot linear case) already introduced. A CNN's layers progress from generic, broadly-reusable low-level detectors to increasingly task-specific combinations, which is exactly why transfer learning works; a Transformer's token embeddings and attention-derived contextual representations play the analogous role for sequences. This unifying thread — the network learns its own features — connects the CNN and Transformer clusters and sets up the final cluster's generative models, which take representation learning one step further: learning a representation compact and complete enough to generate new data from.