中文版: AI 学习路线图 04 · 深度学习

04 · Deep Learning

The AI Learning Roadmap · Stage 4 of 9

Why This Stage Matters

Classical machine learning gave you models that are powerful but hand-crafted: someone decides what the features are. Deep learning removes that bottleneck — instead of engineering features, we train networks that learn the features themselves. A convnet does not need you to hand-design edge detectors; it discovers them from raw pixels. That single shift is why deep learning conquered vision, speech, and language, and why it now underlies every large language model.

The good news from Stage 3: the paradigm is identical. Data → model → loss → gradient → update. Deep learning is the same loop, with bigger models, bigger data, and a few new tricks. This stage makes you fluent in those tricks, and in PyTorch, the framework you will use for the rest of the roadmap.

Core Concepts

1. From perceptron to deep networks

• The neuron is a weighted sum followed by a non-linear activation: output = activation(Wx + b). The weights W and bias b are learned. (You already know this — it's a dot product plus a step, exactly Stage 2.)
• Activation functions provide the non-linearity that lets networks model anything. Without them, stacking layers is just a linear transformation — no more powerful than one layer. The workhorses: ReLU (max(0, x) — default for hidden layers, cheap, avoids vanishing gradients), sigmoid (squashes to 0–1 — probability outputs), tanh (squashes to –1 to 1).
• A multi-layer perceptron (MLP) stacks layers: input → hidden layer → hidden layer → output. Each layer transforms the representation; depth is what lets the network build concepts on top of concepts (edges → shapes → objects).
• Loss functions tell the model how wrong it is: MSE for regression; cross-entropy for classification (paired with softmax — the output layer's probabilities). Picking the right loss is as important as picking the architecture.
• Backpropagation — you met its engine in Stage 2 (the chain rule). Here it becomes an algorithm: compute the loss, then propagate its gradient backward through every layer, giving every weight its own "which direction improves the loss." The framework (PyTorch's autograd) does the calculus; you must understand what it is doing: gradients flowing from output to input.

2. Training dynamics — the practical core

Most of deep learning skill is not architecture; it is training. These are the knobs and techniques you will actually spend your life turning:

• Optimizers: SGD with momentum (momentum = keep a running velocity, don't zigzag) and Adam (adapts each weight's step size from gradient history — the default that "just works").
• Learning rate: the single most important hyperparameter. Too high → divergence; too low → snail's pace. Modern practice: learning-rate schedules (warm up, then decay) and sometimes learning-rate finders (train briefly across rates, pick the steepest descent). A loss curve that plateaus is often a rate problem, not a model problem.
• Batch size: how many examples per gradient update. Small batches = noisy but fast steps (and better generalization, often); large batches = stable but memory-hungry.
• Regularization in deep nets: dropout (randomly disable neurons during training — forces the network to not rely on any single neuron), weight decay (L2 penalty — the deep learning name for what you met in Stage 3), batch normalization (normalize each layer's inputs — stabilizes and accelerates training), early stopping (watch validation loss, stop when it stops improving), and data augmentation (jitter, flip, crop images — free extra data).
• The debugging loop: overfit a single batch first (if the model can't memorize one batch, your code is broken); then watch training/validation curves; when validation diverges from training, you're overfitting — add regularization. Loss curves are your dashboard; learn to read them like an instrument panel.

3. Convolutional neural networks (CNNs) — seeing images

• Why not MLPs for images? An image is a grid of pixels; an MLP treats it as a flat list and loses the spatial structure — a cat shifted by 5 pixels looks completely different to it. Also, it needs astronomically many parameters.
• Convolution fixes both: a small filter (e.g., 3×3) slides over the image, computing dot products at every position. The filter is a learned feature detector (edge, corner, texture); sliding it gives translation invariance — the feature is found anywhere in the image. Key ideas: kernels/filters (learned), channels (many filters → many feature maps), stride & padding, pooling (downsample: max pooling takes the max over a window — shrinks the map, adds a little invariance), and receptive field (deeper layers see larger regions).
• Classic architectures (know the lineage, don't memorize the numbers): LeNet (1998) → AlexNet (2012, the breakthrough) → VGG (deeper, simple 3×3 stacks) → ResNet (2015: residual connections — skip connections that let gradients flow and allow 100+ layers; the idea behind every modern architecture). Today: efficient nets (MobileNet, EfficientNet), and transformers for vision (ViT — Stage 5's architecture applied to images).
• Transfer learning — the most important practical trick. Take a network pretrained on ImageNet (millions of images), keep its learned features, and fine-tune the last layers on your small dataset. With a few hundred images you get near-state-of-the-art results. This pattern — pretrain on huge data, fine-tune on your data — is the deep learning pattern, and it reappears everywhere (BERT, GPT, CLIP). Internalize it now.

4. Sequence models — handling language (the old way)

Before transformers, language was handled with recurrent networks:

• RNNs process text one token at a time, carrying a hidden state forward — the network "remembers" what came before. Problem: the hidden state is a bottleneck; long-range information fades (vanishing gradients).
• LSTMs/GRUs add explicit memory gates so information can survive thousands of steps. For years, these were the state of the art in translation, speech, and text generation.
• Word embeddings — the other pillar: represent each word as a dense vector where similar meanings sit close together ("king − man + woman ≈ queen"). word2vec/GloVe learned this from co-occurrence statistics. The idea — that meaning is geometry — is the foundation of everything in Stage 5.
• seq2seq — encoder reads the input, decoder writes the output (translation). The limitation that matters: recurrence processes tokens one-by-one and struggles with long sequences; and it cannot parallelize. Enter attention — which you'll meet properly in Stage 5.

5. PyTorch — your framework

PyTorch is the language you will speak for the rest of this roadmap. The core objects:

• Tensors — NumPy arrays + GPU acceleration + gradient tracking. x.requires_grad_() starts the autograd recorder.
• Autograd — automatically computes all gradients when you call loss.backward(). You never write backprop by hand.
• nn.Module — every model is a class: layers in __init__, forward pass in forward. Compose like LEGO.
• optim — optimizers: torch.optim.Adam(model.parameters(), lr=1e-3).
• DataLoader — batches, shuffling, parallel loading. Dataset + DataLoader = the data pipeline.
• The training loop (write it a hundred times until it's muscle memory): zero gradients → forward → loss → backward → optimizer step.
• GPU: model.to('cuda') and tensors .to('cuda'); Colab/Kaggle give you free GPUs. (Training deep nets on a CPU is possible for small examples — but a GPU makes the difference between 5 minutes and 5 hours.)

Tools & Skills

• PyTorch (the framework), torchvision (datasets + pretrained models).
• A free GPU — Google Colab or Kaggle notebooks.
• Weights & Biases or plain logs — track experiments (loss, LR, config) so you can compare runs. Even a spreadsheet works; just track.

Hands-On Tasks

1. MLP on MNIST/Fashion-MNIST (week 1). Build the canonical "hello world" of deep learning: classify handwritten digits. Write the full loop: Dataset, DataLoader, model, training, evaluation. Reach ~95%+ accuracy. Try changing the learning rate and watch what happens (log it).
2. Debugging drill (week 1). Deliberately overfit a single batch; then add dropout and watch validation improve. Change batch size and learning rate; record the loss curves. This trains the instinct that separates real practitioners from tutorial-followers.
3. CNN on CIFAR-10 (weeks 2–3). Build a small CNN (conv → ReLU → pool → ... → linear). Watch it struggle, then apply transfer learning: load a pretrained ResNet, freeze the backbone, train the head — and watch accuracy jump. Feel the difference; that difference is why transfer learning dominates practice.
4. Text classifier (week 3–4). Use embeddings + a simple network (or an LSTM) to classify movie reviews (e.g., IMDB). Compare a bag-of-words baseline with embeddings; see what representation buys you.
5. Experiment journal (all weeks). Keep a log: for each experiment, config, curves, result, and what you learned. This journal is your real portfolio of understanding.

Milestone Checkpoint

You are ready for Stage 5 when:

• You trained an image classifier (CNN or fine-tuned) and a text classifier, and you understand every line of the training loop you wrote.
• You can explain backpropagation as "the chain rule applied backward through the network" and say why autograd makes it automatic.
• You can read a loss curve: what does divergence look like? What does overfitting look like? What do you change in each case?
• You can explain why transfer learning works and when to use it.
• You can name the limitation of RNNs that motivates attention.

Curated Resources

• fast.ai Practical Deep Learning (Part 1) — the best hands-on course; you'll build real models in week 1. Free.
• Andrej Karpathy — Neural Networks: Zero to Hero — build everything from scratch; the "micrograd" and "makemore" videos are gold for intuition.
• PyTorch Official Tutorials — "Learn the Basics" and "Quickstart"; then the 60-minute blitz.
• Deep Learning (Goodfellow, Bengio, Courville) — the free reference textbook; read chapters 6 (MLP), 9 (CNN), 10 (sequence) selectively.
• CS231n (Stanford) — the famous vision course notes; the "backprop" and "CNN" sections are the best written explanations anywhere.

Time Estimate & Pace

• Total: ~6 weeks at 10–15 h/week (roughly 75 hours).
• Suggested split: weeks 1–2 → MLP + training dynamics + debugging drills; weeks 3–4 → CNNs + transfer learning; weeks 5–6 → sequence models, embeddings, text classifier, milestone.

Bridge to the Next Stage

Everything you just learned — layers, gradients, transfer learning, embeddings — was preparation for one architecture that changed everything. Stage 5 introduces attention and the Transformer: the machine that reads text not word-by-word but all-at-once, weighs every word against every other, and became the foundation of every modern language model. You now have exactly the right mental toolkit to meet it.