karthikreddy-7/ai-engineering-playbook
Analytics & monitoringA zero-to-100 learning path for applied AI engineering — RAG, embeddings, vector search, agents, MCP, and the production engineering around them. 56 pages, built as a searchable site.
Overview
RAG · embeddings · vector search · agents · MCP · and the production engineering around them Search · dark mode · rendered diagrams · self-graded quizzes · flashcard drills · progress tracking The of AI engineering: you take models you did not train and build systems that are correct, fast, cheap, and safe enough to put in front of real users. Most material on this subject is a pile of definitions. Definitions tell you what something is called. They don't tell you what breaks without it, what it costs, or what you'd reach for instead — and those are the things that matter when you're the one building it. So every page answers three questions, in order: The specific pain that makes the concept necessary, and why the obvious alternatives lose. One level down, with a diagram: the mechanism, the parameters, and the maths that matter. Scale, cost, latency, failure modes — the things that only show up under real traffic.
README
What this is
The consumer side of AI engineering: you take models you did not train and build systems that are correct, fast, cheap, and safe enough to put in front of real users.
Most material on this subject is a pile of definitions. Definitions tell you what something is called. They don’t tell you what breaks without it, what it costs, or what you’d reach for instead — and those are the things that matter when you’re the one building it.
So every page answers three questions, in order:
[!NOTE] This is not a machine-learning course. No PyTorch, no training loops, no backprop. Model internals appear only at survival depth — enough to reason about a boundary question without bluffing.
Who it’s for
Engineers building at the application layer who want to understand these systems properly rather than assemble them from tutorials. It assumes you can code and have called an LLM API.
It’s organized around the questions that decide whether a system works:
- How does RAG actually work — and what does productionizing it involve?
- How does HNSW work, and why cosine similarity?
- How do you evaluate retrieval quality when there’s no single right answer?
- What does document Q&A over 10M documents for 50k users look like?
- When should you not build an agent?
- Where do latency, cost, and failures come from in an LLM pipeline?
The path
Eleven sections in dependency order. Each assumes the ones above it.
| # | Section | What it covers |
|---|---|---|
| 01 | LLM Fundamentals | How LLMs work, tokens & context, sampling, prompting, structured output, APIs, hallucination |
| 02 | Embeddings & Search | Embeddings, similarity metrics, ANN indexes (HNSW/IVF), hybrid search, rerankers |
| 03 | Vector Databases | Choosing a store, metadata filtering, scaling & operations |
| 04 | RAG | The pipeline, chunking, retrieval patterns, graph & agentic RAG, evaluation, production |
| 05 | Agents | The agent loop, tool calling, orchestration & memory, multi-agent, reliability |
| 06 | MCP | The Model Context Protocol, and building MCP servers |
| 07 | Production | Latency, cost, reliability, observability, security, evals & testing |
| 08 | Model Landscape | Choosing models, open source & self-hosting, fine-tuning vs RAG |
| 09 | Multimodal | Vision, audio, and multimodal ingestion at survival depth |
| 10 | System Design | A repeatable reasoning framework + five worked designs |
| 11 | Rapid Fire | Flashcard-style Q→A per section, plus tradeoff one-liners |
How every page is built
| Section | What it does |
|---|---|
| Prerequisites | What the page assumes, linked — so you can walk up when something is shaky |
| The intuition | The idea in plain language, with one load-bearing analogy, before any terminology |
| Why it exists | The specific pain that makes the concept necessary, and why the alternatives lose |
| The core idea | The concept stated properly and densely — the version you’d want to explain out loud |
| How it actually works | One level down, with a diagram: mechanism, parameters, and the maths that matter |
| The flows | The distinct end-to-end paths, each with what selects it and what breaks it |
| A worked example | One concrete input traced through with real numbers — then deliberately broken |
| Production concerns | Scale, cost, latency, failure modes, and what only shows up with real traffic |
| Common drill-downs | The questions a good engineer asks one level down, with answers |
| Test yourself | Scenario questions — reasoning, not recall |
| Go deeper | Curated, individually verified links: official docs, canonical blogs, papers, talks |
| Where this connects | The natural next topics, and why you’d go there |
Every page opens with its prerequisites and closes with where this connects, so the site is a web you can walk rather than a list you read front to back.
Built to be studied, not skimmed
Recognising an answer feels almost identical to being able to produce one — and the two come apart the moment you have to explain something out loud. So the site makes you produce it.
| 🧠 230 quiz questions | Scenario questions at the end of every page. Answer out loud, reveal, then grade yourself. The grade is what counts, not the recognition. |
| 🔁 155 flashcards | The rapid-fire decks in drill mode — keyboard navigation, shuffle, and a weak only filter that comes straight back to what you flagged. |
| 📈 Progress tracking | Mark pages understood and watch the path fill in. Stored in your browser — no account, nothing sent anywhere. |
How to use it
Read the whole path — sections in nav order. Each assumes the ones above it, and pages inside a section are ordered so nothing is used before it’s introduced. Follow the dependency map and you’ll never hit an idea you haven’t met.
Repair one weak spot — start at the page you’re shaky on and walk its Prerequisites links upward until you hit something solid, then come back down.
Fast recall — the rapid-fire decks and tradeoff one-liners. Drill them, say the answer out loud, grade yourself honestly.
The running example
One system recurs across the site so abstract ideas land on something concrete: a governed enterprise LLM platform — a model gateway over a managed cloud model service, with entitlement-filtered retrieval over internal documents, temporary document upload, grounded web search, an agent mode calling internal tools over MCP, and a guardrails/observability layer around all of it.
It’s deliberately generic — an architecture pattern many organizations converge on, not any one company’s system. Numbers used with it are illustrative typical scale, not measured figures.
Correctness policy
- Every external link was fetched and verified before being committed. No dead links, no hallucinated citations.
- Broken internal links, missing anchors, and pages left out of the nav fail the build — they can never reach the site.
- Content is checked against its source on every build, so nothing can be silently lost in publishing.
- This field rots fast. Model names, prices, context limits, and API specifics change monthly. Mechanisms and tradeoffs don’t. Cross-check any specific figure against provider docs before relying on it.
Running it locally
The site is Next.js + Fumadocs, statically exported.
Prose lives in docs/ as plain Markdown; the site content is generated from it.
cd web
pnpm install
pnpm run migrate # docs/*.md -> web/content/docs/*.mdx
pnpm dev # http://localhost:3000
pnpm run check # everything CI enforces
docs/ plain Markdown — the source of truth, edit this
mkdocs.yml the nav order (pedagogical, not alphabetical)
web/ the Next.js + Fumadocs site
content/docs/ GENERATED from docs/ — never edit by hand
components/ quizzes, flashcard decks, progress, diagrams
scripts/ migration, nav generation, and the build checks
Contributing
Spotted an error, a stale claim, or a better canonical resource? Issues and PRs welcome — this field moves fast and correctness beats coverage.
→ Read CONTRIBUTING.md first. The one rule: write in docs/, never edit
web/content/.
License
Install
This server does not publish a one-line install command.
Open the repository installation guide