中文版: AI 学习路线图 07 · 大模型应用工程

07 · LLM Application Engineering

The AI Learning Roadmap · Stage 7 of 9

---

Why This Stage Matters

Stages 0–6 taught you what models are. This stage teaches you what to build with them — and it is where most of the jobs are. Industry is not hiring people to train GPT-sized models; it is hiring people to build reliable systems on top of them: chatbots grounded in company knowledge, assistants that take actions, pipelines that turn documents into answers.

The hard truth that defines this stage: a raw LLM is not a product. It hallucinates, has a knowledge cutoff, ignores your data, costs money per token, and gives different answers to the same question. Application engineering is the craft of building around those limitations — with prompting, retrieval, tool use, evaluation, and disciplined architecture — until what remains is dependable.

Core Concepts

1. Prompt engineering — the first layer

Prompting is not "talking to AI nicely." It is the discipline of eliciting reliable behavior from a model:

• System vs. user messages — the system message sets the permanent rules ("You are a support agent. Answer only from the provided context. If unsure, say you don't know."); the user message is the request. This separation is your first reliability tool.
• Zero-shot vs. few-shot — zero-shot: ask directly. Few-shot: provide 2–5 example input/output pairs so the model pattern-matches your format and style. Few-shot is the cheapest way to improve consistency.
• Chain-of-thought (CoT) — ask the model to reason step by step before answering ("Let's think through this"). For math, logic, and multi-step questions this measurably improves accuracy. Variants: self-consistency (sample several reasonings, vote), and (for structured tasks) "think in private, answer in JSON."
• Structured output — force the model to return JSON (via JSON mode or output schemas) so downstream code can parse reliably. This is how models talk to programs.
• Prompt patterns for reliability — explicit rules ("never invent citations", "if the answer is not in the context, say: I don't know"), delimiters, constraints on length/format.
• When prompting is not enough — prompting changes behavior, not knowledge. If the model needs facts it doesn't have, you need retrieval (RAG). If it needs a new behavior entirely, you may need fine-tuning. Prompting first, always; but know its ceiling.

2. Function calling & tool design — giving the model hands

The model can call functions: it outputs a structured request ("call get_weather(city='Beijing')"), your code executes the function, and the result is fed back into the conversation. This is the mechanism behind every AI agent (Stage 8) and every useful integration today.

Design principles for tools:

• One tool = one well-defined capability with a clear description — the description is the prompt the model reads to decide when to use it.
• Schemas over free text — define parameters as typed JSON; validate before executing.
• Fail gracefully — tools return errors as normal results; the model can read them and adapt.
• Permissions and limits — tools execute real side effects; gate destructive actions (sending mail, writing files) behind confirmation.

3. RAG — retrieval-augmented generation

The problem: your chatbot must answer from your documents, and the model has never seen them — and it hallucinates.

The idea: before generating, retrieve relevant passages from your documents and include them in the prompt: ground the answer in evidence. This is RAG, and it is the single most important pattern in applied LLM work.

The pipeline:

Documents → chunk → embed → store in vector DB
Query     → embed → retrieve top-k → rerank → [context + question] → LLM → grounded answer

Each stage has real design choices:

• Chunking — split documents into pieces (by paragraphs/sections, 200–1000 tokens). Too small = lost context; too big = diluted relevance. Overlap between chunks helps. This is where retrieval quality is won or lost.
• Embeddings — a model that maps text → vector such that similar meanings are near each other. (You know these from Stages 4–5.) Your chunk and query are embedded with the same model.
• Vector databases — store and search embeddings by similarity: FAISS, Chroma, pgvector, Qdrant, Milvus, Pinecone. The key operation is approximate nearest-neighbor search — fast at millions of vectors.
• Retrieval quality — pure vector search misses exact-match/term-match cases; hybrid search (dense + BM25/keyword) usually wins; reranking (a small cross-encoder that re-scores the top-50 with the query) is the highest-leverage quality boost in all of RAG.
• Generation with context — the prompt instructs: "answer only from the context; cite sources; if the context doesn't contain the answer, say so."
• Failure modes — the top of the list: bad chunks (retrieved text is too coarse), missing context (the answer isn't in the retrieved set at all — check recall), "lost in the middle" (the model ignores mid-prompt context), and hallucination beyond the context despite instructions. Diagnose by splitting retrieval quality from generation quality.

The decision framework (you will use this constantly):

Need	Tool
Different style/format	Prompting
Knowledge the model lacks	RAG (fresh, private, or specific data)
A new capability/behavior	Fine-tuning
Taking actions	Function calling → agents
Cost/latency pressure	Smaller model + RAG + caching

RAG first; fine-tune only when RAG + prompting provably fail — it is expensive and frozen.

4. Application architecture — production thinking

A production LLM app is more than a prompt loop. The concerns, in rough priority order:

• Orchestration — how calls are sequenced (retrieval → LLM → validate → maybe more LLM). You can hand-roll it with a few functions, or use a framework (LangChain — batteries included; LangGraph — explicit graphs, the modern choice; or provider SDKs). Prefer the minimal framework that keeps your logic explicit and testable — frameworks age, your understanding shouldn't.
• Caching — identical or similar queries are common; cache exact hits (Redis/DB) and optionally semantic caches (embed the query, return cached answer if a near-duplicate exists). This is usually your biggest cost lever.
• Retries, rate limits, timeouts — API calls fail; design for it (exponential backoff, fallback models).
• Streaming — return tokens as they arrive; it feels 10× faster and users perceive it as responsive.
• Observability — log every call: input, output, tokens, latency, cost, retrieved chunks, model version. You cannot improve what you cannot see. (LangSmith and similar are purpose-built; a structured log table works too.)
• Cost & latency budgeting — measure cost per query and p95 latency from day one; set budgets. Token count is the unit of cost — architecture choices (model size, context size, caching) are cost choices.
• Guardrails — input filtering (prompt injection attempts, abuse), output filtering (PII, unsafe content), and a fallback path when the model misbehaves.

5. Evaluation — the discipline that makes it real

The difference between a demo and a product is evaluation. The practice:

• Build a golden set — 50–200 (start small, grow) representative inputs with expected behaviors: the answer, the format, and what "good" looks like.
• Automate scoring — for structured tasks, check programmatically (format validity, required fields, retrieved-context recall). For open answers, use LLM-as-judge: a strong model rates outputs against a rubric (correctness, faithfulness to context, tone). Validate your judge against human labels on a sample first.
• Regression testing — every change (prompt, chunking, model, retrieval) reruns the golden set. A change that improves one case and breaks three is a regression, not an improvement.
• Retrieval metrics separate from generation — measure retrieval recall@k and answer faithfulness separately, so you know where failures live.
• A/B in production — the golden set is the lab; production metrics (resolution rate, user satisfaction, cost/query) are the field. Both matter.

Tools & Skills

• An LLM API (any provider) and an open model for fallback/experiments.
• A vector database — start with Chroma or FAISS (local, simple), graduate to pgvector/Qdrant.
• LangGraph or LangChain — or your own minimal orchestration; pick deliberately.
• An eval setup — golden set + LLM-as-judge script (or LangSmith/Evals frameworks).
• Observability — structured logging from day one.

Hands-On Tasks

1. Prompt drill (week 1). Take one task (e.g., extract structured info from a messy text). Iterate: zero-shot → few-shot → system rules → JSON output. Measure correctness on 10 test inputs at each step. Feel the difference engineering makes.
2. Function calling (week 1–2). Build a tiny "assistant" with 2–3 tools (e.g., calculator, weather mock, web search mock). Have the model decide which tool to call, call it, and feed the result back. This is the skeleton of every agent.
3. RAG — the milestone (weeks 2–4). Pick a real corpus (your own documents, a wiki dump, or public docs). Build the full pipeline: chunk → embed → store → retrieve → rerank → generate. Then evaluate: build 20–30 questions, measure retrieval recall and answer faithfulness, and iterate on chunking and retrieval until scores are high. Add hybrid search and reranking; measure the delta.
4. Cost & latency audit (week 4). Measure your RAG app's cost per query and p95 latency. Add exact caching; measure the improvement. This is what "production thinking" means.
5. Golden-set regression (week 4–5). Turn your eval questions into a scripted suite. Make a prompt change; run the suite; report regressions. You are now doing evaluation like a professional.

Milestone Checkpoint

You are ready for Stage 8 when:

• You built a RAG chatbot over a real corpus with measured retrieval and answer quality.
• You have a golden set + automated eval that reruns on every change.
• You can explain the prompting vs. RAG vs. fine-tuning decision framework.
• You know your app's cost per query and can name its three biggest cost drivers.
• You implemented at least one guardrail and one cache, and you know why they exist.

Curated Resources

• DeepLearning.AI short courses — "LangChain for LLM Application Development", "Building Systems with the ChatGPT API", "Building and Evaluating Advanced RAG" — 1–2 hours each, exactly this stage in practice.
• Anthropic — "Building effective agents" — the clearest industry essay on when and how to build agents; read it now, re-read at Stage 8.
• Pinecone Learning Center — the best free RAG deep-dives (chunking strategies, hybrid search, reranking).
• LangGraph documentation & tutorials — if you choose the framework; its tutorials double as architecture patterns.
• Building LLMs for Production (O'Reilly) — a full book on production concerns (evaluation, cost, deployment) when you want depth.

Time Estimate & Pace

• Total: ~6 weeks at 10–15 h/week (roughly 75 hours).
• Suggested split: week 1 → prompting + function calling (tasks 1–2); weeks 2–3 → RAG build (task 3); week 4 → eval, cost, caching (tasks 4–5); weeks 5–6 → polish, production concerns, milestone.

Bridge to the Next Stage

Your application answers questions grounded in data and can call tools. Stage 8 removes the last constraint: instead of answering one query per request, the system will pursue multi-step goals — planning, remembering, calling tools repeatedly, and acting until the job is done. That is an agent, and it is where this whole roadmap has been pointing.