中文版: AI 学习路线图 05 · NLP 与 Transformer

05 · NLP & The Transformer

The AI Learning Roadmap · Stage 5 of 9

---

Why This Stage Matters

Language is the interface to most of modern AI. Every chatbot, every search engine with an answer box, every coding assistant — they all rest on one architecture: the Transformer, introduced in the 2017 paper "Attention Is All You Need." If you understand the Transformer, you understand the engine of the entire LLM era; if you don't, everything from Stage 6 onward is magic you can't debug.

The great news: by now you already know almost everything the Transformer needs. You know embeddings (words as vectors), you know layers and gradients, you know transfer learning. This stage adds exactly one genuinely new idea — attention — and then shows you how it assembles into the architecture behind BERT, GPT, and everything since.

Core Concepts

1. Why language is hard (and how we cheat)

Text is discrete, variable-length, and deeply contextual. "Bank" means something different next to "river" than next to "money." Old approaches (bags of words, n-grams) ignored order or exploded combinatorially. The modern pipeline handles this in three steps:

• Tokenization — split text into tokens. Word-level splits miss morphological variants ("run", "runs", "running"); modern subword tokenizers (BPE, WordPiece, SentencePiece) split into frequent subword units: tokenization → token + ization. This keeps vocabulary manageable (~50k tokens) and handles any word, including invented ones. You will use Hugging Face tokenizers constantly; the important thing is that token IDs, not characters, are the model's alphabet.
• Embeddings — map each token ID to a dense vector. The key property, learned from data: similar meanings live close together, and relationships become vector arithmetic (king − man + woman ≈ queen). This is distributional semantics: a word is characterized by the company it keeps. (You met word2vec in Stage 4; now it becomes the input layer of everything.)
• Positional encoding — a Transformer processes all tokens simultaneously, so it has no inherent sense of order; we inject position information into each token's representation so "dog bites man" ≠ "man bites dog."

2. Attention: the one new idea

Here is the problem that killed RNNs: to understand the word "it" in "The cat chased the mouse because it was hungry," the model must connect "it" to "cat" across distance — and RNNs can't carry information reliably that far.

Attention is a weighted lookup over the whole context. For each token, the model looks at every other token and decides, how much should I attend to each one? Concretely, every token produces three vectors:

• Query (Q) — what am I looking for?
• Key (K) — what do I contain?
• Value (V) — what do I actually contribute?

Then: each query scores every key (dot product = similarity), the scores become weights (softmax), and the token's new representation is a weighted sum of the values. In plain English: "I ask my question, see which other words are most relevant, and blend their information in proportion to relevance."

Two refinements matter:

• Multi-head attention — run attention several times in parallel with different learned projections, so the model can attend to different kinds of relationships at once (syntax, coreference, sentiment, position). Each head is a different "lens."
• Self-attention — every token attends to every token in the same sequence. This is what gives Transformers their two superpowers: long-range dependencies (any two tokens are one step apart) and parallelism (all tokens processed at once, unlike RNNs — which is why they scale to billions of parameters).

The cost: attention is O(n²) in sequence length (every token attends to every token). For a 1,000-token input that's a million pairs — manageable; for 100,000 tokens it's a problem you will meet again in Stage 6.

3. The Transformer architecture

A Transformer is a stack of identical blocks, each block being:

Token embedding + positional encoding
        │
   [ Self-attention ] ──┐
        │               │ residual connection (+)
   [ Layer norm ]       │
        │               │
   [ Feed-forward ]  ───┤
        │               │
   [ Layer norm ]       │
        │               │
        ▼               ▼
   next block (same shape)

Two structural habits explain the whole design:

• Residual connections — each sublayer is x + sublayer(x) (you met these in ResNet). Gradients flow freely, so stacks can be 100+ layers deep. It's how deep learning scales in practice.
• Layer normalization — stabilizes training (the modern cousin of batch norm for sequences).

Encoder vs. decoder — the fork that splits the family tree:

• Encoder-only (BERT, RoBERTa): reads text bidirectionally — every token attends to every other, left and right. Great at understanding: classification, question answering, retrieval.
• Decoder-only (GPT, Llama, all modern LLMs): generates left-to-right; each token attends only to what came before (causal masking — otherwise the model could "cheat" by seeing the answer). Great at generation.
• Encoder-decoder (T5, BART): encoder reads input, decoder generates output. Classic for translation and summarization.

The modern landscape note: decoder-only models won. Everything from GPT-4 to open models (Llama, Mistral, Qwen, DeepSeek) is a decoder trained to predict the next token. When people say "LLM" today, they mean a scaled-up decoder.

4. Pretraining and fine-tuning — the recipe

The Transformer's other revolution is how it's trained, in two phases:

• Pretraining — train on enormous unlabeled text with a self-supervised objective:
  • BERT: masked language modeling — hide 15% of tokens, learn to predict them (fill-in-the-blank). The model must understand both sides of context.
  • GPT: next-token prediction — given the prefix, predict the next token, over and over, on trillions of tokens.
  • No human labels needed — the text itself is the label. This is why scale is possible: the internet is the dataset.
• Fine-tuning — take the pretrained model and continue training on a small, labeled dataset for your task (sentiment, classification, summarization). This is transfer learning, exactly as you did with ResNet in Stage 4 — pretrain on huge data, adapt on small data.

The result is the pattern that defines modern NLP: one pretrained model, infinite tasks. And a bonus ability you will meet in Stage 6 — in-context learning: a large enough pretrained model can often do a new task with just a few examples in the prompt, no weight updates at all.

5. The Hugging Face ecosystem

Hugging Face is the standard toolchain, and its beauty is consistency: every model, whatever the architecture, shares the same three APIs.

• tokenizer — text → input IDs (tokenizer("Hello!")), with padding/truncation built in.
• model — input IDs → predictions; AutoModelForSequenceClassification, AutoModelForCausalLM, etc.
• pipeline — one line for a full task: pipeline("sentiment-analysis")("I love this!").
• The Hub — thousands of pretrained models and datasets, one search away. This is where you download BERT, GPT-2, Llama, and everything else.

Tools & Skills

• Hugging Face Transformers — the library (pip install transformers).
• Hugging Face Datasets — loading standard datasets (IMDB, GLUE, etc.).
• A GPU (Colab/Kaggle free tier) for fine-tuning.
• Optionally: Weights & Biases to track fine-tuning runs.

Hands-On Tasks

1. Tokenizer exploration (day 1). Load a BPE tokenizer; tokenize a sentence in English and one in Chinese; inspect the token IDs; decode them back. See how subwords handle morphology and how vocabulary size affects tokens-per-word.
2. Visualize attention (day 2–3). Load a small pretrained model and visualize attention maps for a sentence — the original "Attention is All You Need" paper has the classic diagrams; bertviz makes it interactive. Watch "it" attend to "cat." This is the moment the architecture becomes real.
3. Fine-tune BERT (week 2). Fine-tune bert-base-uncased for sentiment classification on IMDB (or any classification dataset) with the Hugging Face Trainer. Reach ~90%+ accuracy. Compare with your Stage 4 text classifier: same task, but now the model was pretrained on all of English first.
4. Generate with GPT-2 (week 2–3). Load a small decoder model (GPT-2 or a modern small model); generate text; experiment with temperature (higher = wilder) and top-p sampling; watch repetition, a real failure mode, up close.
5. Summarization (week 3). Use an encoder-decoder model (e.g., BART or T5) via pipeline("summarization") on a long article; see length control and how summaries degrade on very long inputs (a preview of context-length problems).

Milestone Checkpoint

You are ready for Stage 6 when:

• You fine-tuned a pretrained Transformer for classification and it clearly beat your Stage 4 model.
• You can explain, in your own words, Q, K, V — what each is "asking" and "answering."
• You can explain the difference between encoder-only, decoder-only, and encoder-decoder, and name a model of each type.
• You can explain the difference between pretraining and fine-tuning, and why pretraining on raw text is possible without labels.
• You generated text and can name two failure modes you observed.

Curated Resources

• Jay Alammar — The Illustrated Transformer — the best visual explanation ever written; read it twice.
• Hugging Face NLP Course — the practical bible; chapters 1–7 cover everything in this stage hands-on.
• The Annotated Transformer (Harvard) — the paper, line by line, in runnable PyTorch. Go as deep as you like.
• Attention Is All You Need (Vaswani et al., 2017) — the original paper; after the Illustrated Transformer it will make sense.
• Andrej Karpathy — Let's build GPT — build a GPT from scratch in ~2 hours; the definitive "I actually understand this now" moment.

Time Estimate & Pace

• Total: ~4 weeks at 10–15 h/week (roughly 50 hours).
• Suggested split: week 1 → tokenization, embeddings, attention (tasks 1–2); week 2 → architecture + fine-tune BERT (task 3); week 3 → generation + summarization (tasks 4–5); week 4 → review, read the paper, milestone.

Bridge to the Next Stage

You now understand the engine. Stage 6 zooms out to the machine built on it: what happens when you scale this architecture to billions of parameters and train on most of the internet — the capabilities that appear, the alignment techniques that tame them, how these models are served in production, and how to evaluate them honestly. You will go from "how does a transformer work" to "what is an LLM, really."