Evaluation Pitfalls: Noise, Cheating & Regression

LearnAI0Learn AI from zero.

Topic 3/Evaluation Pitfalls: Noise, Cheating & Regression

0 / 384

Search lessonsLearning focus

Core courseRecommended order

01

Before You Start

Find your starting point and understand how to study AI before entering the main course.

Where Are We? The Dunning-Kruger EffectUsing the Dunning-Kruger Effect to analyze the three stages of AI learning: Peak of Mount Stupid, Valley of Despair, and Plateau of Sustainability. This course helps you truly understand what AI is doing, not just how to use it.How to Learn So Knowledge Actually SticksWatching ≠ Learning. The right approach: pause after each example to reflect, map it to your own business context, then output to validate. This three-step loop is how knowledge actually sticks.Why Do We Spend So Much Time on Fundamentals?Without understanding the underlying principles, your AI product is built on guesswork. Grasping the core mechanisms of large language models lets you pinpoint issues precisely and choose the right solutions.

03

AI Foundations

Build the first mental model for what AI can do, why it fails, and how to talk to it.

Skip the Theory for Now — Look at the Amazing Things It Can DoWhat can AI actually do? Turn meeting recordings into minutes, make sense of medical reports, explain how planes fly to a 6-year-old, coach you through interview practice, build a webpage without knowing how to code… 6 interactive demos show you: AI is unlike any software you've ever used — we used to learn the software; now the software listens to us.What Exactly Is AI? It's Really Just Playing "Finish the Sentence"What exactly is AI? Why can ChatGPT, DeepSeek, and Doubao chat with you? A plain-language explanation for total beginners: AI isn't really thinking — it's a 'super sentence-finishing machine' that has read an enormous amount of text. A 3-minute interactive demo to build your first correct intuition about AI.It Isn't Searching the Web for Your AnswerMany people assume AI is just a smarter search engine — it's a completely different thing. A search engine finds webpages other people wrote; AI answers directly from 'the memory of what it has read'. Once you understand this difference, you'll know when to search, when to ask AI, and which claims are worth double-checking.It Spouts Nonsense with a Straight FaceAI invents books that don't exist, gets health facts wrong, and fabricates celebrity quotes — in exactly the same tone it uses when telling the truth. This is 'AI hallucination'. 3 spot-the-fake mini-games let you catch AI making things up with your own hands, plus three practical little habits to take away.

Treat It Like a Brilliant New Colleague Who Knows Nothing About YouWhy do other people get amazing results from AI while your questions get mediocre answers? Because AI is a brilliant 'new colleague' who knows absolutely nothing about you. A three-level question comparison + a prompt building-block assembler, teaching the 'context + request + constraints' formula.The Universal Opener: "Ask Me a Few Questions First, Then Start"Can't write prompts? No problem. One sentence — 'before you start, ask me a few questions' — can double the quality of AI's answers, by handing the puzzle of 'I don't know what to specify' back to the AI. An interactive dialogue demo lets you experience the power of question-first prompting firsthand.

How Do You Actually Write a Good Prompt?How do you actually write a good prompt? No templates to memorize: just cover three things clearly — background, request, and constraints — and you'll already beat nine out of ten prompts. Plus three handy tricks: give an example, let the AI interview you first, and point at exactly what you don't like. From the Beginner FAQ series.Is "Prompt Engineering" Only for Engineers?Is Prompt Engineering something only engineers need to learn? For everyday users, it's the craft of saying things clearly. For people building products, it's a job description written into a system and called millions of times. Understand this difference and you'll see why it's a real craft.Models, Agents, AI Apps — How Do They Relate?How do large models, Agents, and AI apps relate to each other? One analogy makes it clear: the model is the engine, the Agent is the full car with wheels and a steering wheel, and the AI app is the ride-share you hail on your phone. Get these three layers and AI news will finally make sense.What Makes Agents So Powerful? Why Everyone's Talking About ThemWhat makes Agents so powerful? A chat AI gives you a method; an Agent gives you the result: it breaks down tasks, uses tools, checks its own work, and retries when something goes wrong. One expense-report example explains why everyone's talking about Agents in 2026.What Is This "Skill" Everyone's Talking About?What is this "Skill" everyone's talking about? It's simply a cheat sheet of experience written for AI: a plain text file that tells the AI what to do first, then next, when it meets a certain kind of task. It's not new technology, but it turns an AI that comes back to ask you 4 times into one that finishes everything in a single trip. Play the animation and see the difference for yourself.What Is Vibe Coding? Can You Build Software Without Writing Code?You keep seeing posts like 'I built an app with one sentence.' That practice is called Vibe Coding: you direct AI in plain language, and it writes the code. You bring the request and the feedback. Can a non-programmer really ship a useful little tool? Play the replay, then we'll talk about the limits — and where to start.What Chinese AI Models Are Out There? How to Choose?What Chinese AI models are out there? Qwen (Tongyi), Kimi, DeepSeek, Doubao, Zhipu GLM, ERNIE, Hunyuan, MiniMax… each one shines at something different. Click your scenario to find your match. Data based on the LMArena global blind-test leaderboard (August 2026).Beyond OpenAI, Which AI Companies Matter?Beyond OpenAI, which AI companies matter? Anthropic's Claude, Google's Gemini, Meta, Musk's Grok, France's Mistral… First play a matching game to pair each company with its flagship product, then browse a friendly company field guide.

What Is a Token? Why Does AI Charge by It?AI charges by token — so what is a token? It's the smallest unit AI uses to read and write text, billed like a taxi by distance: every token read or written costs money. Try the tokenizer on this page and watch a sentence get sliced into tokens as the bill jumps.Why Does It "Forget" After a Long Chat?Something you already said, and mid-chat the AI forgets? Each answer can only see the most recent content that fits on the desk. When the desk is full, the earliest words get pushed off. Drag the slider and watch the turn-1 'peanut allergy' note fall off.What Are "Reasoning Models" and "Deep Thinking"?A lot of AI apps hide a Deep Thinking switch. Turn it on and answers get slower and the bill gets bigger — so what are you actually paying for? A reasoning model works the problem out on scratch paper first, then hands in the answer, and it gets hard problems wrong less often. Same seating puzzle, two modes: watch instant vs. deep thinking side by side.Does More Parameters Mean Smarter?The news keeps comparing who has more parameters. What even is a parameter? Think of it as the number of knobs in a brain: more knobs usually means broader knowledge, but how smart it is also depends on training data and methods. Drag the scale slider — from the 1-billion tier to the trillion tier — and feel the difference.Why Can't Some AIs "See" Pictures?Why can some AIs see images while others only say 'please describe it in words'? Because 'speaking' and 'seeing' are two independent skills: language models natively only understand text, and seeing pictures requires an extra pair of 'eyes' (a vision encoder). An interactive experiment explains multimodality.What Is Fine-Tuning? How Is It Different from "Feeding It Docs"?Fine-tuning sends the AI back to class and drills knowledge into its body. Feeding it docs (a knowledge base, RAG) is an open-book exam — look it up as needed. One takes weeks and costs tens of thousands; the other is ready the same day, and you update it by swapping a file. Play the side-by-side animation, then match four scenarios to see which one your need actually calls for.What Is This "Knowledge Base" Every Company Is Building?What is this "knowledge base" every company is building? Slice company files into chunks, build an index, and before the AI answers it first finds the relevant passages, stuffs them into the conversation, then replies — and it can cite the source for you. Play the three-step animation, then compare how differently the AI answers the same question with and without a knowledge base.GPT, LLM, AIGC… How Do You Tell These Acronyms Apart?GPT, LLM, AIGC, AGI, NLP… how do you actually tell the AI acronyms in the news apart? AI is the biggest circle, LLM is the large model inside it that handles text, GPT is a mainstream way to build LLMs, and AIGC is the act of using AI to generate content. Play a matching game — finish it and you get a who-contains-whom map.Why Is NVIDIA Worth So Much?Why is NVIDIA worth so much? AI training has to run a huge volume of simple arithmetic at once, and GPUs are built for that human-wave approach: a CPU is like one PhD grinding through problems one by one; a GPU is like ten thousand elementary-school kids starting at the same time. The whole world is scrambling for compute, and NVIDIA is the biggest shovel seller in this gold rush. Start the arithmetic race and see the gap with your own eyes.

"I Trained a Model" — What Did They Actually Train?Someone at dinner always says they 'trained a model' — what did they actually do? Training means changing the model's parameters, and it costs from hundreds of thousands to over a hundred million yuan. Most people wrote a prompt or attached a knowledge base. A four-rung ladder tells the layers apart; three follow-ups let you hear which layer they're on.AI Jargon TranslatorWhat do “we built our own LLM,” “deeply empower,” and “All in AI” actually mean? The same word in AI circles can point to completely different things. This page has 8 talking-point cards: tap to translate into plain language, with a gold-content rating, plus three follow-ups that get the real story on the spot.Does an “Open-Source Model” Mean It's Free?Is an open-source model free? Most 'open source' only publishes the weights — training data and methods usually stay closed. Download costs nothing, but running it needs a GPU and electricity, and commercial use still depends on the license. This page uses a three-piece checker and a full-size vs. distilled comparison to show what 'open source' really means in AI.Why Does the "#1 on the Leaderboard" Model Feel Worse in Real Use?Why does the #1 model on the leaderboard feel worse than a lower-scoring one? Because the board tests an exam, and you need it to do work: the question bank can be gamed, and the syllabus may not include your scenario. This page uses a reversal demo to show three traps of benchmark scores, then shows you how to build a 10-question personal quiz to pick a model.The More We Chat, the Better It Knows Me — Is It Learning?The more you chat, the better AI seems to know you — is it secretly learning? The model's brain was frozen the moment it left the factory. Chat doesn't change the model itself. What we call memory is the product writing your details in a little notebook and stuffing them back in before each conversation. Open the animation and see how far the picture in your head is from what actually happens.When an AI Detector Says "This Was Written by AI," Can You Trust It?Your paper gets flagged as "AI-generated," your resume gets marked red by a screening tool — can you trust an AI detector? Sadly, it's closer to mysticism: AI learned by studying good human writing, so people who write cleanly get falsely accused. Play a short judgment game, feel how unreliable the detector is, then pick up a few ways to prove you wrote it.Are "Secret Prompt Playbooks" Worth Buying?Is a few-hundred-yuan 'secret prompt playbook' worth buying? Probably not: the skeleton that actually works fits on half a page, and this site already teaches it for free. The rest is practicing and revising on your own jobs. Stamp through six talking points on a typical flyer — these playbooks mostly sell anxiety.Why Is the Answer Different Every Time?Ask AI the same question twice and the answers often differ. Bad memory, or is it blowing you off? Neither: every time AI writes a word, it rolls the dice and draws from a few high-probability candidates. That's an intentional design so answers stay natural, not stiff. There's a dice animation on this page — roll it a few times and you'll get it.Why Is AI Customer Service So Dumb?In the same era, chat AI is as sharp as a good colleague, but the store's support bot still answers the wrong question. It's dumb for a reason: the merchant wants to save money, the guardrails are locked tight for fear of saying something they'll have to pay for, and a pile of old keyword bots are still operating under an AI sticker. Plus a few tricks to reach a human fast — tap through and compare for yourself.Are Siri and ChatGPT the Same Thing?They're both called AI assistants — so why has Siri spent a decade saying "Sorry, I didn't catch that," while ChatGPT can talk about anything? The difference is the technical path: old-school voice assistants run on command matching, like a phone keypad menu; generative AI is like a live operator. Play the animation and watch the same sentence take two different paths.

Is Free AI Enough? When Is Paying Worth It?Is free AI enough? The free tier usually gives you a model one size smaller, plus a usage cap — plenty if you only dip in now and then. This page takes the free tier apart: model tier, quota, context, and peak-hour queues. Then it helps you find your match and decide when paying is worth it.What Is an API? How Is It Different from a Membership?What is an API? How is it different from a monthly membership? A membership is an all-you-can-eat buffet — eat whatever you want in their restaurant. An API is loose ingredients sold by the gram — you take them home and cook. Drag the usage slider to see which way of paying is cheaper for you, then we'll unpack what “topping up a key” actually means.Why Does Generating One Image Cost Dozens of Times More Than a Chat?Why does generating one AI image cost dozens of times more than a chat? A text exchange costs a fraction of a cent; one image costs a few cents. Three reasons: images carry far more data than text, they're 'painted' dozens of times over, and the GPU is monopolized the whole time. An animated comparison makes it intuitive.Why Is AI Video Generation So Expensive?Why is AI video generation so expensive? Because video = twenty-plus coherent images per second + sound + physical plausibility. Drag the slider and see: a 10-second video means generating two hundred-plus frames, each one staying consistent with the rest. Pricing based on mainstream 2026 video model APIs.What Are API Relay Resellers? Why We Don't Really Recommend ThemWhat is an API relay reseller? Why is it cheap? Why don't we really recommend it? Not that all sellers are bad actors — but the structure itself carries three unavoidable risks: your data passes through strangers, the model may be quietly swapped, and the service can vanish overnight. A route animation makes it clear.What Are "Reverse-Proxy Accounts" and "Shared Accounts"?What are 'ChatGPT Plus carpool for ¥9.9/month' and 'Claude shared accounts'? A reverse-proxy account is one real account split among dozens of people. Others may see your chat history, the account can be banned at any moment, and the password sits in someone else's hands. A 'shared apartment' animation explains the risks.Will My Chat History Be Used for Training?Will my chat history be used for training? It might — it depends on the product and your settings. Quite a few products use your conversations to improve the model by default, but legitimate products all give you a switch to turn it off. A data-flow diagram shows where your words go, which kind of settings hide the switch, and what you should never send to AI.

Which Jobs You Can Hand Over with Confidence, and Which Deserve One Extra CheckNaming things and writing copy can be safely handed to AI; calculating taxes or asking about medication doses are better double-checked — but how do you decide quickly? An 8-question quiz game + two little questions: does this have one correct answer? How bad is it if the answer is wrong? An AI-trust judgment method anyone can use.All Six Intuitions Are in Hand — Where to Next?Wrapping up the AI basics chapter: a self-check list of six core intuitions (finishing sentences, not a search engine, it makes things up, the new-colleague mindset, ask-me-first, the trust quadrants), then 3 questions to find the learning path that fits you best.

04

Large Language Models

Connect training data, tokens, GPT-style models, and hallucinations into one foundation.

AI's Food: Training DataExplore the scale and quality of LLM training data: trillion-Token corpora, data cleaning pipelines, Common Crawl / Books / code composition ratios — and how data scale determines a model's knowledge ceiling.Training vs. Inference: Two Completely Different ProcessesTraining and inference are two completely different stages of a large language model. A detailed breakdown of gradient descent and weight updates during training, and autoregressive generation during inference, to help you understand their fundamental differences and resource costs.Vocabulary & TrainingA deep dive into how large model vocabularies are built: BPE tokenization algorithm, the relationship between Tokens and words, Chinese vs. English Token efficiency, and the meaning of word-vector matrices — the first step to understanding LLM generation.Base Model: The Token-by-Token Prediction MachineWhat is the core of large language models? The Base model learns statistical patterns of Token sequences from massive corpora. Each generation does one thing: predict the most probable next Token. Understanding this mechanism is where everything begins.GPT's LeapFrom GPT-1 to GPT-4, how does Scaling Law drive emergent capabilities? Why did the pre-training paradigm outcompete all task-specific models? A deep dive into the mechanics behind the great leap in large language models.

The Mystery of chat/completionsWhy is OpenAI's chat/completions endpoint called 'completions'? Understanding the true nature of the conversational API: every request is an independent text completion; multi-turn conversation is assembled client-side, not stored server-side.A language model only knows how to complete text — so what if I fabricate a chat log and ask it to finish it?How did OpenAI transform GPT from a completion engine into a conversational assistant? By faking chat logs to teach the model role-playing — the very origin of instruction alignment.Chat Template + SFT: Large Models Finally Learn to "Talk"How does Supervised Fine-Tuning (SFT) turn a Base model into a Chat model? A deep dive into the design of Chat Template, the role of System/User/Assistant markers, and the basics of RLHF.Why Are Prompts Enough? The Answer Lies in the Context WindowEach inference of a large language model can only see content within the context window. Understanding the context window mechanism explains why a well-crafted Prompt can make a world of difference in model performance.

LLM HallucinationWhat is LLM Hallucination and why does it occur? Starting from the statistical nature of Token prediction, this lesson analyzes the root causes of hallucination and four mitigation strategies: Prompt constraints, RAG, Temperature tuning, and human evaluation.Prompt Engineering: Constraining Hallucinations with InstructionsA well-designed System Prompt can significantly reduce hallucination rates in large language models. Covers prompt design patterns such as defining knowledge boundaries, requiring citations, and refusing uncertain answers.RAG: Let the Model "Take an Open-Book Exam"RAG (Retrieval-Augmented Generation) is the most widely used engineering solution to reduce LLM hallucinations. A deep dive into how RAG works: vector retrieval, context injection, and grounded generation.RAG Costs & Optimization StrategiesRAG reduces hallucinations but comes with real costs: increased latency, doubled Token consumption, and retrieval quality that directly impacts answer accuracy. Deep dive into RAG optimization strategies: chunking, reranking, and hybrid search.Mitigation 3: Temperature & Top-PTemperature and Top-P are the core parameters for controlling LLM output randomness. Intuitive demos explain: Temperature=0 makes the model more deterministic; higher values increase divergence; Top-P limits the candidate Token pool.Mitigation 4: Evaluation + Human ReviewAutomated evaluation and Human-in-the-Loop review are the last line of defense for AI product quality. Learn how to design evaluation metrics, build test sets, and embed human review into AI products.

Summary (Part 1) · What is an LLM + HallucinationChapter 1 key takeaways: Token generation, Base model principles, GPT milestones, Chat Template, and the nature of the context window — building your LLM mental model.Summary (Part B) · Mitigation Strategies + Decision FrameworkChapter 1 Part B summary: comparing four hallucination mitigation approaches (Prompt, RAG, Temperature, Evaluation), plus the foundational mental model every AI PM should build.

Don't stop at "I get it." Lock That One Job Down firstDon't stop at 'I get it' after LLM fundamentals. This page walks the job you want to hand to AI down to a requirement you can accept: lock input and output, define what counts as right, pick a hallucination response by risk — plus three tasks you can start today at 15 min / 1 hour / half a day.

LLM Fundamentals · 30 Must-Answer QuestionsChapter 1 interview practice: how will an interviewer, your boss, and a tech colleague test your LLM fundamentals? 30 real-scenario questions, each with the examiner's intent, an answer framework, and bonus points: probability prediction, message list, hallucination explanation, RAG vs retraining, Temperature, context window.

05

AI Harness

Learn context engineering, prompting, safety, agents, tool use, and cost-aware model choices.

Context Window: AI's Working MemoryA deep dive into how the context window works in large language models: why it acts as AI's working memory, how Token budgets are allocated, performance degradation with long contexts, and the core ideas of context engineering.Context Overflow: Three Handling StrategiesWhat do you do when a conversation exceeds the context window? Compare three strategies: Truncation, Summarization, and RAG (vector retrieval), analyzing their respective use cases and trade-offs.

Why LLMs Choose MarkdownWhy do LLMs naturally prefer Markdown output? Analyzing the deep relationship between Markdown and large language models from three angles: training data distribution, Token efficiency, and structured semantic expression — to guide Prompt design.Markdown Syntax and Engineering RenderingHow to handle Markdown output in AI products? Covers common Markdown syntax, frontend rendering library selection, streaming rendering timing, and UX optimization.Tell It What to Be, and It Becomes ThatCore principles of Prompt engineering: LLMs have no genuine understanding, only pattern matching. How to precisely control model output quality and style through role assignment, task decomposition, few-shot examples, and constraints.Advanced Prompt TechniquesMaster advanced Prompt techniques: Chain of Thought for step-by-step reasoning, Self-Consistency for more reliable answers, and structured output templates for machine-readable results.Output Format Trade-offsWhich output format should you choose for different scenarios? Compare JSON (programmatic parsing), Markdown (user display), and plain text (voice/simple scenarios), plus engineering approaches for enforcing structured output.Streaming Output & Format CoordinationStreaming is the key to AI product UX. This page explains SSE protocol fundamentals, how streaming output pairs with Markdown rendering, and the challenges and solutions of streaming JSON parsing.

Prompt Injection: Why Do Attacks Succeed?Prompt Injection is one of the most serious security threats facing LLM-based applications. A deep dive into the mechanics of injection attacks: how attackers override System Prompt instructions via user input.Prompt Injection: 12 Attack Cases12 real-world validated Prompt injection attack cases: direct injection, indirect injection, role-play bypass, ignore instructions, Base64 encoding bypass, and more — to build security awareness for AI product managers.Prompt Defense: Three-Layer Interception in PracticeHow do you defend against Prompt injection attacks in real products? This lesson covers multi-layer defense strategies including input filtering, instruction separation, defensive System Prompt design, and output leak detection — using the MoodVerse case as a foundation.AI Safety Red Lines: Four Non-Negotiable PrinciplesFour non-negotiable AI safety red lines for enterprises: sensitive data stays inside, credentials never enter conversations, high-risk operations require human confirmation, approve before use and label after. Follow these four rules to avoid 90% of AI security incidents.Risk Tiers & AccountabilityAI usage risk tiers (L1/L2/L3) and accountability chains (User/Approver/Manager): different risk levels require different controls, and every role is responsible for their own AI usage.

Agent: AI That Gets Things DoneWhat is an AI Agent? Four core capabilities — Plan / Tool / Memory / Act — rooted in Lilian Weng's profoundly influential architecture, each with step-by-step demonstrations in real-world scenarios.The Secret of Tool CallsFunction Calling (tool calls) is a core capability of Agents. A deep dive into the OpenAI Function Calling protocol format, how to write tool descriptions, how models decide which tool to call, and how to handle tool results.5 Messages Behind One ConversationA complete anatomy of Function Calling: the left side shows the user-facing conversation; the right side shows the actual Message List transmitted via the API. Each step is annotated with design decisions for product managers.The Art of Tool DescriptionsTool description quality directly determines AI tool call success rates. Through comparison experiments, demonstrates the vast difference between good and bad descriptions. Product managers should be involved in designing tool descriptions.Multi-Tool Orchestration: Concurrent vs. SequentialMulti-tool orchestration: concurrent or sequential? Visualize and compare serial, concurrent, and smart-batch tool execution strategies with a Gantt chart, helping product managers understand key framework-level design decisions.MCP Protocol: The USB Interface for AI ToolsMCP (Model Context Protocol) is the USB standard for AI tools. Tool developers write one adapter and connect to all Agents.ReAct in Practice: When the Agent Gets StuckThe ReAct (Reasoning + Acting) framework lets Agents reason and act while executing. Real-world case analysis of the ReAct iterative loop: Thought, Action, Observation, plus common causes and solutions for Agents getting stuck.Short-Term Memory = Context WindowThe context window is the model's only short-term memory. Through a visual workspace demo, see how 128K Tokens are quickly consumed across multi-turn conversations, and the design decisions product managers must make.Context Compression: Four Layers of DefenseContext compression four-layer defense: 60% Snip, 75% MicroCompact, 85% Collapse, 95% AutoCompact.Long-term Memory: Vector RetrievalLong-term memory lets the Agent remember last month: through vector retrieval, inject user preferences, project config, and historical bugs into the current conversation for truly personalized service.From Embeddings to MilvusUnderstand embeddings, semantic similarity, ANN, and the role Milvus plays.The Milvus Mental ModelLearn Collection, Schema, Entity, Index, Search, Query, and Load through a familiar database mental model.Hands-on MilvusConnect, create a schema, batch insert, index, load, search, query, and delete with MilvusClient.From Retrieval to RAGConnect Milvus retrieval to RAG with chunking, filtering, reranking, hybrid search, and RRF.ReAct Loop: Think → Act → ObserveA detailed breakdown of the ReAct loop: how an Agent completes complex tasks through iterative Think-Act-Observe cycles. A single task of removing console.log calls can take 6+ loops.5 Patterns of Agent DeadlocksAgents are not infallible — they have 5 typical failure patterns. Good product design confronts these issues and prepares a response plan for each failure mode.Permissions & SafetyAgent permission and safety design: interactive comparison of Confirm, Auto, and Smart modes. Product managers must balance safety and efficiency.Skill: Getting Agents to the Destination FasterSkill is a high-level encapsulation of commonly used Agent tool chains that reduces the decision-making burden. Covers Skill design principles, differences from Tools, and how to build a reusable skill library.The Essence of SkillSkill turns Agent exploration into execution: compare the efficiency gap when an Agent performs the same task with and without a Skill. Product Managers need to decide which operations to pre-install Skills for.Dissecting a Real-World Skill Scaffolding Engineering: From Prototype to ProductionAgent Harness (Scaffolding) is the key to moving experimental Agents into production. Covers retry mechanisms, error recovery, tool timeouts, log tracing, and human-in-the-loop Harness design patterns.5 Engineering GuardrailsAn Agent's core capability comes from the LLM, but its reliability comes from engineering guardrails. Iteration cap, output truncation, timeout control, interrupt recovery, and context emergency—5 guardrails take an Agent from usable to production-ready.Multi-Agent Collaboration: When One Agent Isn't Enough, Deploy a TeamMulti-Agent collaboration: when a task is too complex for one Agent, deploy a team. The main Agent breaks down tasks, dispatches sub-Agents, and aggregates results. Read-only tasks run in parallel; write tasks run in series.Observability: What Is the Agent Doing and How Much Is It Costing?Observability in practice: use event streams to see in real time what the Agent is thinking, doing, and spending. Essential operational infrastructure for product managers.Agent Engineering LandscapeAgent Engineering Landscape: A complete knowledge map of the four pillars — Tools, Memory, Execution, and Engineering. Click any node to see details.

Why Do Multi-Turn Conversations Get More Expensive?LLM APIs charge by Token. In multi-turn conversations, each turn must send the full history, causing Token consumption to grow linearly. A visual breakdown of multi-turn Token billing logic to help you understand why costs explode.KV Cache: Trading Space for SpeedKV Cache is the core technique for accelerating large model inference. How it works: caching the K/V pairs of Attention to avoid redundant computation on existing tokens, simultaneously reducing latency and cost.Explicit Cache: Practical ComparisonHow to use Anthropic's Prompt Caching and OpenAI's explicit cache API? By fixing the System Prompt prefix and ordering messages correctly, guarantee KV Cache hits on every request.Dynamic Timestamps: The Most Expensive System PromptInserting the current timestamp into the System Prompt is one of the most common and costly AI development anti-patterns. It prevents KV Cache from ever hitting, forcing full recomputation on every request.Comprehensive Cost OptimizationAI system cost optimization is more than compressing Prompts. A five-layer comprehensive optimization framework — from model selection and request merging to caching, output truncation, batching, and traffic control — helps AI product managers make data-driven cost decisions.Image Tokens: pixels cost money tooHow do multimodal models charge for images? This explainer breaks down image Token calculation: images are sliced into tiles by resolution, each tile consuming a fixed number of Tokens. Learn to optimize multimodal costs by controlling image dimensions.Image Tokens: match resolution to taskDifferent tasks have vastly different resolution requirements: OCR needs high resolution, while sentiment analysis works fine at low resolution. Choose the right image resolution for your task and avoid paying for unnecessary pixels.Syntax-Level Optimization: Prompts for Machines — Skip the FormattingHow to reduce token consumption at the syntax level of Prompts? Compare token efficiency of natural language vs. structured instructions, and learn practical techniques like removing redundant words, using XML tag delimiters, and choosing token-efficient formats.Semantic-Layer Optimization: Don't Treat the Context Window as a Trash CanThe context window is a precious computational resource — don't fill it with irrelevant information. Explains semantic-layer strategies: how to identify and remove redundant history, dynamic summary compression, and keeping only high-relevance content.Output Layer + Advanced KV Cache: Control Output = Control CostOutput tokens typically cost 2–3× more than input tokens. This lesson covers how to control output costs via length limits, conciseness requirements, and format constraints, plus advanced KV Cache scenarios and boundary conditions.Model Selection: Trade-offs of Capability vs. CostHow to estimate costs before launching an AI product? Covers Token usage estimation methods, cost calculation under concurrent models, model price comparisons, and a practical framework for setting cost budgets.Simplicity Is the Ultimate Sophistication: Upholding First PrinciplesAll the complexity of AI engineering ultimately reduces to a few first principles. Starting from first-principles thinking, distill the core insight of AI Harness: simplicity, measurability, iterability.

Human–AI Knowledge Boundaries: The Four-Quadrant StrategyWhat to delegate to AI, what to do yourself — a quick four-quadrant task allocation framework Good vs Bad Prompts: Context Is EverythingContext determines output quality; same question, good vs bad side-by-side demo Can You Trust What AI Says?Three quick verification methods; spotting when AI confidently gets it wrong The Art of Iteration: Knowing When to StopPractical AI tips: The Art of Iteration. Through a 4-round iteration demo of marketing copy, learn to judge when to keep refining and when to stop. Solving problems in three rounds is expert-level; stopping at round three is master-level.Scenario Guide: When to Trust AIAI usage scenario guide: 9 common scenarios ranked into three risk tiers — safe, caution, and high-risk. Email/brainstorming/translation are safe; data analysis and coding need verification; law, medicine, and investing are hallucination danger zones.

Summary (Part 1) · Prompt Engineering + AgentChapter 2 first-half summary: context engineering, advanced Prompt techniques, Markdown usage guidelines, Prompt injection attack and defense, and core Agent design patterns.Summary (Part B) · Cost Optimization + PM PerspectiveChapter 2 second-half summary: a five-layer cost optimization framework (KV Cache, Prompt compression, output control, model routing, architecture optimization), and the core decision-making capabilities of an AI PM at the Harness stage.Course Summary · An AI Primer for Product ManagersCourse summary for An AI Primer for Product Managers: a one-page overview of the four modules — LLM Fundamentals, Prompt Engineering, Agent & Cost Awareness, Safety & Compliance — covering core concepts and common misconceptions.Summary (Part 1) · Fundamentals + HarnessCourse summary, part 1: Module 1 — LLM fundamentals (frozen parameters, hallucination as a structural problem, RAG) and Module 2 — Prompt engineering (System Prompt, Few-Shot, CoT) key concepts.Summary (Part 2) · Practice + CostA Product Manager's First AI Course summary, Part 2: Module 3 — Agent & Cost Awareness (Plan+Tool+Memory+Act, five-layer cost optimization framework); Module 4 — Security & Compliance (Prompt injection, three-layer defense, product red lines).

Write one prompt. Make Five Runs in a Row All UsableAfter context engineering, advanced Prompt work, and Agent design, this page takes the M0 requirement and writes it as a first-version prompt, checks stability with five runs in a row, then welds it shut with format constraints and Few-Shot examples. Three tasks you can start today.

AI Harness · 30 Essential QuestionsHarness core interview practice: How will an interviewer, boss, and tech colleague test you on AI Harness? 30 real-scenario questions, each with assessment intent, answer framework, and bonus points: context overflow, Prompt engineering, injection defense, Agent and tool calls, Skill, multi-Agent, KV Cache and five-layer cost optimization, model selection.

06

From Demo to Product

Follow a real agent product from working demo to usable AI application.

A Real Agent Build LogUse a real agent desktop app as context for later practical lessons.

Text-to-Image vs. Image-to-Image: Two Completely Different ThingsUsing AI to Write Prompts for AIThe user says one sentence; the LLM translates it into the detailed Prompt the image model needs. This two-stage translation architecture is the core design of image generation products.Character Consistency: The Hardest Product Problem When the Model Goes Down, Then What?Image Generation Productization Checklist

Textbook's 3 Steps vs. Real-World N StepsTextbooks describe ReAct as three steps: Think→Act→Observe. But in production, each loop involves far more than 3 steps.Why Agents Get Stuck in Infinite LoopsSeveral typical failure patterns of Agent loops in real production environments, and what users actually experience when things go wrong.Foolproof Design: How to Make the Loop Stop ItselfThree types of protection strategies to safely stop Agent loops: hard limits as a last resort, detection and early warning, and graceful degradation.Streaming UX: Don't Leave Users WaitingWhen a tool runs in the background for 30 seconds, what should the user see on screen? Progress-sense design is the key to Agent product experience.The Real Cost Behind a Single MessageA single user message can trigger 10+ loops and dozens of API calls under the hood. Visualize the real token consumption and costs.

Longer Conversations Cost More and Get DumberThree reasons context management is mandatory: escalating cost, declining attention, and a hard window limit. Conversation is not a free-flowing stream.Compression Is the Art of Trade-offsSome can be deleted, some cannot, some require paid compression: context compression is about finding balance between information value and token cost.Can You Delete What the User Said?User messages are sacred: better to delete 1,000 words of AI output than to touch 10 words from the user.Local Compression vs. LLM CompressionCompress as much as possible with free methods first; only pay for AI help when you've exhausted local options. A guide to choosing between the two compression approaches.

Context ≠ MemoryContext is AI's scratch pad — close the session and it's gone; memory is AI's notebook — it's still there next time you open it.What's Worth Remembering and What Isn'tIf you remember everything, the memory store quickly turns into a landfill. A gatekeeper matters more than memory capacity.Memory Conflict: What to Do When the User Changes Their MindPeople change their minds. An AI memory system must do more than append — it must learn to update.The Cost Problem of Memory InjectionMore memory isn't always better. Stuffing 1,000 entries into every prompt makes it harder for AI to find what matters.

System Prompt Is Not a Blob of TextA production-grade System Prompt is layered: Identity, Environment, Tool, and Behavior layers — each with its own responsibility, none interfering with the others.Don't Show AI What It Doesn't NeedStuffing 100 tool descriptions into the system prompt can take up tens of thousands of tokens. Lazy loading: provide only names on the first turn, expand descriptions on demand.Skill: An Operatable Prompt ModuleA Skill is a file containing the Prompt and tool usage instructions for a specific task. File-as-configuration, version-traceable.The Subtle Relationship Between Prompts and CachingChange a single character in the System Prompt and the entire KV Cache is invalidated. How you inject dynamic content directly affects cache hit rate and cost.

When Do You Need Multiple AgentsThree real-world scenarios where one Agent isn't enough: parallel acceleration, role separation, and risk isolation. Most of the time, one is sufficient.The Cost of Concurrency: Who Can Run in ParallelReads can be parallel, writes must queue. How to determine whether a tool operation can run concurrently.Brainstorm: Let Multiple AIs ArgueBrainstorm mode: throw the same question to multiple Agents for independent thinking, then the moderator summarizes consensus and divergence.The Cost Trap of Scheduled TasksWhen an Agent runs scheduled tasks by reusing old sessions, the context grows longer each time and costs snowball. The correct approach: create a new session every time.

How Much Freedom Should AI HaveThe spectrum of five permission modes: from fully autonomous to step-by-step approval. There is no best permission mode — only the one most appropriate for the current scenario.Too Many Popups Frustrate Users; No Popups Means No SafetyBalancing Human-in-the-Loop: constant popups frustrate users, but zero popups shifts all blame. The solution is risk-level grading.Do You Know What Your Agent Did?What tools did the Agent call in the background? How many tokens did it consume? Were there any errors? Observability is the foundation of production AI.

MCP Is More Than Just "Calling Tools"The MCP protocol works in two directions: AI as a Client calling external tools, and AI as a Server being called by Cursor and others. One protocol, bidirectional integration.Lazy Connect: Don't Connect Until NeededRegistered 10 MCP services and connecting all of them on startup? Lazy connect philosophy: registration ≠ connection; connect only when first used.AI Adds Its Own ToolsWhen an Agent discovers it needs a new tool, it can configure the MCP connection on its own. The self-configuration concept: AI manages its own toolbox, but needs user approval.

Practicum PanoramaA panoramic map connecting all topics in the Practicum module: image generation, loops, context, memory, Prompt, multi-Agent, safety, and MCP.Chat Wrapper vs a True Agent ProductA chat wrapper has one loop and one model; a true Agent product has an eight-dimensional Harness. The difference isn't in the code — it's in product decisions.

From “can talk” to “can do”: wire up the first toolYou've watched a real product go from demo to launch. This page is about wiring the first tool onto your own Agent: how to pick it, how to write the tool description, and how to run the full request–validate–execute–feed-back loop. Three task tiers you can start today.

Practice · From Demo to Production · 30 Soul-Searching QuestionsChapter 3 practical assessment: how will your interviewer, boss, and tech colleagues test your AI Agent productization knowledge? 30 real-scenario questions, each with examining intent, answer framework, and bonus points: Demo-to-production gap, Agent loop control, context compression, long-term memory, multi-Agent collaboration, MCP ecosystem, cost control.

07

AI Engineering Patterns

Study production agent patterns for context, tools, evaluation, long runs, and safety.

Workflow vs Agent: Know What You Need FirstUnderstand the core difference between Workflow and Agent: when to use predefined execution paths, and when to let the LLM make autonomous decisions.Five Workflow PatternsA detailed guide to five proven Workflow design patterns: Prompt Chaining, Routing, Parallelization, Orchestrator-Workers, and Evaluator-Optimizer.From Prompt Engineering to Context EngineeringFrom optimizing prompts to curating the entire context window — understand why Context Engineering is the core capability of the Agent era.Three Strategies for Long-Context AgentsThree core strategies for handling long-task context challenges: Compaction, Structured Note-taking, and Sub-agent Architecture.

ACI: Agent-Computer InterfaceAgent-Computer Interface (ACI) is a core concept in Agent design: tool design deserves as much investment as HCI. Learn the four principles of tool design, poka-yoke (mistake-proofing), and the art of writing tool descriptions.Milvus as an Agent Knowledge ToolWrap Milvus as a search_knowledge tool and test when an Agent should and should not call it.Think Tool: Making AI Think Before ActingThe Think Tool is a special tool with no side effects — it simply lets an Agent pause and think mid-action. In complex tool chains and policy-intensive scenarios, it boosts accuracy by 54%.Using an Agent to Optimize Agent ToolsA three-step workflow of using Claude Code to write tools, run evaluations, and automatically optimize tools, plus five core tool design principles. Tool quality determines the ceiling of Agent quality.

Why Evaluation Matters More Than TrainingExplains why evaluation is the most underrated phase in Agent development: building an Agent without evals is like flying blind.Three Grader Types: Code, Model, HumanThe pros, cons, and combination strategies of three Grader types in Agent evaluation: Code Grader, Model Grader (LLM-as-Judge), and Human Grader.Evaluation Pitfalls: Noise, Cheating & RegressionThe three most common evaluation pitfalls: infrastructure noise, Eval Awareness (model recognizes it's being tested), and Prompt changes causing regression. Based on real production cases.Current

Why Agents Fail at Long TasksWhy do Agents always fail at long tasks? Two core failure modes: one-shotting everything and premature completion.Initializer + Coding AgentSolve the handoff problem in long-running Agents with the dual-role Initializer + Coding Agent pattern: progress files, feature checklists, and incremental commits.Managed Agent: Brain-Hand SeparationManaged Agent architecture: separating Session, Harness (brain), and Sandbox (hand) so each component can fail and be replaced independently.Session Is Not the Same as Context WindowSession is the Agent's hard drive; Context Window is RAM. Understanding the difference is essential to designing long-running Agents that never lose state.

Three Risk Categories: Misuse, Misbehavior, External AttackUnderstand the three core risks facing Agents: user misuse, model misbehavior, external attacks, and the dual Model-Layer + Environment-Layer Containment strategy.Sandbox & Credential IsolationA deep dive into Agent sandbox isolation and credential management: why generated code and secrets must never coexist in the same place.

Contextual Retrieval: Better RAGUnderstand how Contextual Retrieval solves the context-loss problem in traditional RAG by prepending a context prefix to each Chunk, reducing retrieval failure rate by 67%.Advanced OverviewA panoramic review of the advanced chapter: seven topics spanning design patterns, context engineering, tool design, and security architecture—forming a complete Agent knowledge system.Build the Simplest Thing That WorksThree core lessons distilled from the Claude Code source: the heart of an Agent is state management; a Harness encodes assumptions; as models grow stronger, your engineering gets simpler.

Stop saying “it feels better”: build your first eval setAfter Anthropic's design patterns and eval methodology, this page is about building your first eval set: 10 real cases, a pass criterion for each, and a baseline score. From then on, prompt changes aren't based on vibes — better or worse, one run tells you.

AI Engineering Design Patterns · 30 Essential QuestionsChapter 4 interview practice: How will an interviewer, boss, and tech colleague test you on AI engineering design patterns? 30 real-scenario questions with assessment intent, answer framework, and bonus points: context engineering, long-task three pillars, grep vs RAG, tool design ACI, evaluation methodology, sandbox and credential isolation.

08

Harness and Self-Improvement

Understand how harnesses can improve workflows and eventually improve themselves.

From Scaffolding to an Self-Improving SystemUnderstanding the core concept of Recursive Self-Improvement (RSI): Harness is more than scaffolding — it is becoming the optimization target of AI self-improvement.Three Design Patterns of HarnessDeep dive into three Harness design patterns: workflow automation, file system as persistent memory, and sub-agents with background jobs — based on Lilian Weng's Harness Engineering for Self-Improvement.

Context Engineering: From Handcrafted Prompts to Auto-Evolving SystemsFrom ACE to MCE to Meta-Harness: context management is evolving from handcrafted prompts to auto-evolving code systems.Workflow Design: From Manual to Automated SearchFrom AI Scientist to AFlow: workflow design is no longer a manual craft — it is a search space that can be optimized algorithmically.

Letting the Harness Improve ItselfLetting an Agent improve its own Harness: the STOP recursive optimizer and Self-Harness propose-evaluate-accept loop.Evolutionary Search: Letting the Fittest Harness SurviveFrom AlphaEvolve to DGM: using evolutionary algorithms to search the Harness design space and let the fittest Agent survive.

Future Challenges: Seven Hurdles of Self-ImprovementFrom weak evaluators to reward hacking, from diversity collapse to the human role, seven future challenges facing Harness self-improvement.

It's sharp for ten turns. What about fifty?Harness self-improvement sounds far off, but you can build the prerequisite today: a compression threshold, a never-delete list, and notes written to disk. This page helps you find the turn your Agent starts forgetting, and keep important conclusions alive across sessions.

Harness & Self-Improvement · 30 Soul-Searching QuestionsChapter Five in practice: how will interviewers, bosses, and tech colleagues test you on Harness and self-improvement? 30 real-scenario questions, each with the intent behind it, an answer framework, and bonus points: Harness definition, three key design patterns, autonomous context engineering evolution, and the principles and risks of recursive self-improvement.

09

Vibe Coding Methodology

Build a reusable AI collaboration workflow around goals, context, acceptance, safety, and docs.

Why You Need to Set Rules for AIVibe Coding Methodology Lesson 1: AI writes code fast, but without rules it easily reworks, misses edits, and breaks things. Rules are the most reliable way to inject context.

Four-Step Flow: Restate, PRD, Confirm, CodeVibe Coding methodology session 2: the four-step flow of restate, PRD, confirmation, and coding — plus bulk-edit breakpoints and duplicate-wheel detection.PlayGround: The Component Fitting RoomVibe Coding methodology lesson 3: the PlayGround component page — a lightweight Storybook approach. Build an isolated demo first, then integrate.

Style Convergence: Don't Ship Eight CSS Kits for One ButtonThe most typical tech debt in a Vibe Coding project: eight CSS kits for the same button. This lesson covers why styles proliferate, how to fold them into tokens and shared components, and which differences should stay.Three Comment Elements and Code ProtectionVibe Coding Methodology Lesson 4: The three comment elements (context, design intent, key constraints), comment protection, code deletion declarations, dependency change declarations, and error handling standards.The Debugging Iron Rule: Log First, Then FixVibe Coding methodology section 5: add logs first to locate the root cause before changing code, no guesswork fixes, answer three questions before fixing, declare impact scope after fixing.No Partial DeliveriesVibe Coding Methodology Lesson 6: No simplified implementations; no partial deliveries accepted. The real reason AI loves "ship a simple version first" — and how to break the pattern.

Three Documents & Methodology CaptureVibe Coding Methodology, Lesson 7: FEATURES, CHANGELOG, and RELEASE_NOTES each cover a different dimension; METHODOLOGY.md preserves product reasoning. Includes a document triage mini-game and a status-transition demo.

Encode Environment Facts in RulesVibe Coding methodology, lesson 8: Hard-code environment facts such as model configuration, tech stack, and data formats into Rules. Includes a live isComposing input experience area and a data-format tripartite selector.Three Safety Gates for Destructive OperationsVibe Coding Methodology Lesson 9: Back up the database first, provide a rollback plan before irreversible operations, and run a diff review before deploying. Includes a release diff review simulator and a three-gate checklist.

Long-Conversation Anchoring and Writing StandardsVibe Coding Methodology Lesson 10: Long-conversation context anchoring — restate goals every 10 turns. Writing style rules and a forbidden-pattern checklist. Two interactive demos: Context Drift Simulator and AI-Tone Detector.

The Value of Rules: Every Rule Solves a Real ProblemVibe Coding Methodology finale: interactive panoramic map of all 14 rule chapters, a four-step adaptation self-audit wizard (Delete/Swap/Tune/Add), and an 11-lesson recap. The value of each rule is that it solves one real problem.

Last square: turn "how you do it" into "anyone can do it"Last square on the main track: take the pits you hit and the rules you set in the first five squares, and distill them into your own AI collaboration spec. Write your first Rule, run the four-step flow once, export a complete Build Log — from 'I made it' to 'anyone can reproduce it.'

Vibe Coding Methodology · 30 Soul-Searching QuestionsChapter 7 practical quiz: How will interviewers, managers, and dev colleagues test your Vibe Coding methodology? 30 real-world questions, each with the examiner's intent, answer framework, and bonus points: setting AI rules, the 4-step flow, quality baselines, refusing phased delivery, doc retention, and the three gates for destructive operations.

10

Taste Engineering

Train the ability to judge and describe visual quality when working with AI.

Execution Is Free. Judgment Just Got ExpensiveOnce AI knocked down the barrier to shipping, anyone can make something usable in ten minutes. Usable starts to devalue, and taste becomes the new dividing line. Vote on three AI-generated booking pages first, then drag the scarcity scale and feel how judgment gets more expensive.Where the AI Look Comes FromAsk AI to whip up a page and nine times out of ten you get the purple-gradient, frosted-glass, centered-headline trio. Tap five AI-look tells on a typical AI page, then twist the specificity knob and watch how prompts pull AI off the average.

Hierarchy: One Hero per ScreenAttention is scarce. Emphasize everything and you emphasize nothing. Pull the three levers yourself—type size, weight, and color: A/B dashboard spot-the-difference, a lever lab, and cut-the-extras quizzes. Learn one hero per screen.Whitespace & Alignment: Most Ugliness Is SpacingWhen you can't name what's ugly, the problem is usually spacing. Two rules—proximity and consistency—plus an 8pt grid. Breathing-room slider, settings-page alignment spot-the-difference, pricing-card A/B duel: feel whitespace and alignment yourself.Restraint: Budget Your Colors and TypeBudget your colors and type: one primary plus neutrals, two font weights, one radius set. Tap Add a Color five times to watch the page degrade, then one-click subtract to bring the premium feel back; the budget panel scores live, and a quiz breaks down where cheapness comes from.Consistency: Where System Feel Comes FromEach piece looks fine alone; together they feel cobbled-together — what's missing is a shared set of variables. Using this site's real top-bar CSS variables, learn design tokens: a variable switch for the mismatched feel, one radius variable driving five controls, and a two-screen spot-the-difference.

A Hundred Concrete Ways to Say "Looks Good"Tell AI "make it look better" and it can only guess. Say "raise information density one notch, cut visual noise" and it knows exactly what to change. Twelve taste-vocabulary cards with do/don't cases, a complaint-to-term matching game, plus a copy-ready taste vocabulary cheat sheet.Generating UI: From Adjectives to a Spec“Make a nice landing page” vs “primary #DB2777, 8pt spacing grid, no gradients” — that's a whole tier apart. Compare three prompt levels, then assemble a generating-UI spec you can copy.Generating Images: The Language of Composition, Light and Shadow, ColorThe aesthetic trio for image prompts: composition (rule of thirds, whitespace, leading lines), light and shadow (side light, golden hour, contrast), color (low saturation, unified palette, accent color). Train your eye on three real-image pick-ones, accept one of four poster candidates, plus a copyable image-prompt block builder.

What It Means to Really Look at DesignBookmarking a hundred sites equals bookmarking none. To really look, take it apart in a fixed order: hierarchy first, then spacing, then colors, then details. The in-page X-ray lens offers four views so you can dissect a landing page live, with a four-step teardown checklist.Feeding References to AIDescribe style in words and nine out of ten warp. Feed references straight to AI and land it in one shot. How to choose among reference images, style descriptions, and design variables—with scenario matching, copyable prompt templates, and an A/B duel.

Acceptance Checklist: Run Through This Before You Sign OffTaste Engineering finale: AI drafts in ten minutes; three minutes of acceptance is a bargain. A 12-item checkable acceptance checklist reins in 'good enough,' then revisit the opening vote to see if your eye leveled up—with a full-chapter recap and further reading.

11

Interaction Engineering

Improve AI-built interfaces with states, error prevention, controls, and flows.

It Runs. Now What?AI's default interaction ships at the「it runs」level: blank empty states, alert popups for errors, deletes with no confirm. Flip Cooper's considerate-software checklist and you get a spot-the-flaws list. Click through five inconsiderate spots in a todo app that runs fine.

The Three States: Loading, Empty, ErrorThe three abnormal moments in a UI are exactly where experience splits: empty states should teach the next step and offer examples; loading should say how far you've gotten; error states must cover what happened, why, and what to do. Pick a winner in three A/B duels, then rewrite three weak error lines into plain language.Error Prevention & Reversibility: So Users Aren't Afraid to ClickRather than report errors after the fact, stop them from happening: disable what's unavailable, keep dangerous actions at a distance, and ship safe defaults. Undo makes exploration safe; confirm dialogs are the weakest defense. Delete a contact yourself, compare a confirm dialog vs an undo toast, then run a cry-wolf experiment.Flow Restraint: Every Extra Step Drops Another BatchActions that don't directly serve the goal are excise: navigation, interruption, and re-entry are the three big sources. Trim a five-step signup to two yourself, watch completions in the funnel double, then drag a slider to see how many drop with each extra step.Conventions: Don't Let AI Invent New InteractionsUsers operate software through conventions — learn once, use everywhere. Affordance makes controls self-explain. Spot three clever inventions in an AI-generated settings page: a progress-ring Save button, a link that needs a double-click, and a sideways-scrolling form — then see why anything that looks like a button must be clickable.

Did You Pick the Right Control: Radio, Checkbox, Toggle & DropdownMutually exclusive pick-one uses a radio button; multi-select uses checkboxes; instant effect uses a toggle; dropdown only when options overflow one screen. Dialogs interrupt — skip them when you can. Six match-up scenes, dropdown vs flat duel, five should-it-pop quizzes, plus a control cheat sheet you can paste into prompts.The UI Speaks: How Users Read Your Copy"Delete these 3" is more honest than "OK": button verbs must name the consequence, labels use the user's words, don't serve database field names to users, and OK/Cancel pairs are lazy. Rewrite three lines in a dialog and watch the mock update live, then translate five system phrases into human talk.

Goal-Oriented: Describe Requirements by Goals, Not Feature ListsFeed AI a feature list and it returns a counter; feed it a scenario with a persona and a goal, and it returns a tool that fits. Duel two prompts, sort six goal-vs-task lines, run a greedy-persona experiment, then rewrite a feature list into a scenario by hand.Put the State Machine and Boundaries in the PromptAI only writes the states you name; anything you skip, it invents as the happy path. A state-machine completer fills a fair-weather prompt into five states, an edge-case wheel draws six cases AI loves to gloss over, then a block builder assembles an interaction spec you can copy.

Interaction Acceptance Checklist: The Second List Before LaunchInteraction Engineering finale: a 14-item checkable interaction acceptance checklist covering states, error prevention, flow, conventions, control copy, and considerateness; a page hiding five interaction accidents as a drill; three pre-launch lists in relay, plus chapter recap and further reading.

12

AI Product Psychology

Design how users perceive speed, trust, errors, value, and long-term AI product use.

Engineering metrics pass. Why do users still call it slow?AI Product Psychology lesson 1: the server's stopwatch and the user's stopwatch don't tick the same. Four playable experiments: the same 5-second request races blank freeze, spinner, streaming, and visible steps side by side—pick the worst one; feel out 8 seconds and see how much no-feedback waiting gets overestimated; drag the TTFT slider to replay typing and watch first-token time trade for feel score; three timing quirks, one tier at a time. Perceived performance is what users score you on.

The Psychology of Waiting: It Was Never Really About Those 5 SecondsThe Psychology of Waiting: Maister's three laws of waiting with Disney queue boards and the Houston airport baggage case, plus Nielsen's three response-time tiers translated into AI product decisions. Five playable demos: a duration slider for presentation tier, side-by-side law comparisons, waiting design for a 25-second Agent task, background-notification walkthrough, and an engineering-cost quiz.Labor Illusion: Make AI Show Its WorkHarvard Business School research by Buell and Norton on the labor illusion: sites that show their work make users wait 30–60 seconds longer—and raise satisfaction. Five playable experiments: instant reply vs. visible-process A/B vote, recreate the flight-search study yourself, flip three switches on a visible-process assembler and watch trust rise, fake-progress spot-the-difference to catch trust bombs, and three boundary questions on when to perform and when to hold back.Peak-End Rule: Users Only Remember the Peak and the EndKahneman's peak-end rule: the remembered score of an experience is set by the peak and the end, not much by the average or the duration. Five playable experiments: four session plots that show how average score and memory score diverge, a cold-water experiment where you guess which trial most subjects would redo, an experience-curve editor where you build a curve with a negative average but a maxed-out memory score, a ¥100 model-budget allocator, and an Agent wrap-up duel. Engineering takeaway: spend the flagship model on first impression and the last step, put validation up front, and persist intermediate artifacts.

Trust Calibration: The Best Users Are Half-SkepticalTrust calibration: an AI product’s trust target was never meant to be a perfect score. Users who trust everything paste hallucinations into court filings; users who trust nothing turn AI into an expensive paperweight. Four playable labs: judge trust in six real scenarios, click open citations to catch the one that doesn’t match the source, tune wording/citations/warnings in a confidence-signal designer, and watch a constant disclaimer go invisible across five answers in a Boy-Who-Cried-Wolf demo.Algorithm Aversion: One Mistake and the AI Gets Permanently BlockedAlgorithm aversion: Dietvorst’s experiment shows people abandon an algorithm after seeing it err once—even when it outperforms humans overall—while forgiving a person for the same mistake. Cast a vote to catch your own bias, then try the attribution translator, the tweak-permission switch, and the abandonment-rate dashboard: flip four engineering levers and watch the needle move.Defensiveness: Users Aren’t Unable — They’re Afraid to Use ItThree kinds of user defensiveness in AI products: data defensiveness (afraid to upload contracts), competence defensiveness (fear that using AI makes you look replaceable), and responsibility defensiveness (who takes the blame when it errs). Five hands-on toys: six quick questions to map your own defensiveness profile, three levers flipped one by one, draft state plus one-click undo to feel reversibility, three high-defensiveness redesigns (watch for fake switches), and a flip-card on the hand-off-to-human button paradox.

Mental Models: Users Brought the Wrong ManualMost AI one-star reviews come from mental-model mismatch: users treat AI as a search engine (ask for sources, get fabrications), a database (ask for precision, get probabilities), a learning apprentice (“I told you and you didn’t listen”), or a calculator (it even fails at sums). Spot the wrong manual in four real dialogues, then play three correction toys: an empty-state designer that turns the first screen into a classroom, a “Remembered” visibility duel that ends “won’t listen,” and a boundary-first duel that shows how timing turns a disclaimer honest or defensive.How Far to Anthropomorphize, and the Art of AI ApologyCASA paradigm (Reeves & Nass, The Media Equation): people automatically apply social rules to machines—users will treat AI as some kind of person; you only pick the level. Five hands-on experiments: politeness quiz, anthropomorphism-level slider watching expectation and ethics risk rise, seat five product types, apology three-elements assembler that builds a fire-killing apology, four apology copy styles plus the service recovery paradox.

Honeymoon Cliff: Hype Raises Expectation—Retention Pays It BackExpectation-confirmation theory: satisfaction equals experience minus expectation—every notch you hype, the product owes one more back. Play the gacha to see “demo is P99, users get P50,” drag the hype slider to watch signup conversion and 30-day retention trade off, then flip three levers on the expectation-curve editor: promise one notch below, unlock capability gradually, bank improvements into felt releases.AI Label Discount: Same Content, Mark It AI and It Drops in ValueAI label discount: the same content gets systematically lower ratings once labeled AI-generated; people who use AI at work also fear colleagues seeing them—that’s usage shame. Run a double-blind rating experiment to feel the discount, judge five scenes on whether the label must show or can stay off, drag a wording ladder from “AI-generated” to “AI-assisted” to see how deep the discount goes, then pick the export page users dare to share.Cognitive Offloading: Users Rely on You—and Fear Going RustyCognitive offloading: the Google effect shows people drop from memory what they can look up; AI expands what can be outsourced from recall to thinking itself. Users rely on you while fearing they’re going rusty—and that anxiety hits renewals and referrals. First pick an outsourcing checklist to see which offloads are risky, then drag the positioning slider from Copilot to Autopilot, and flip three switches to turn AI answers from “think for you” into “think with you.”Emotional Attachment: After Users Fall in Love with Your ProductUsers form real emotional attachment to AI—the Replika 2023 incident proved how deep it goes and how risky it is. Grade six lifelike user messages with the attachment-signal classifier, open the four-act Replika timeline, then flip three safety valves—identity reminder, vulnerable-topic handoff, impermanence disclosure—and see how they fit into a companion product.

The Psychology of Paying: Where It Hurts to Pay for a Probabilistic GoodAI is a probabilistic good: same money paid, one generation shines and the next flops. Play the metering-anxiety simulator—feel how usage billing makes every follow-up sting, then switch to monthly and watch the mood of the same chat change; work four mental-accounting questions; flip three quota-design levers and see free quota turn from a cost sink into a converter.The Psychology of Pricing: Anchors, Decoys, and Decent Price DiscriminationPrice has no objective feel—it's all reference points. Recreate The Economist's decoy experiment: remove the tier nobody picks and sales flip; flip three switches to build your own pricing-page anchors and watch mid-tier conversion move; then judge which of five forms of price discrimination are decent and which are career-ending; AI products get natural versioning via model tiers; finally pick the price-hike email that won't get you roasted.

The Psychology of Feedback: Why Users Don’t Thumbs-DownA thousand users dislike an AI answer; only a dozen thumbs-down. Play the feedback-funnel simulator and watch silence bias eat your signal layer by layer—flip three switches to widen the funnel; then read six behaviors as implicit signals: regenerate, copy, edit—behavior doesn’t lie; finally duel two post-thumbs-down experiences: feedback needs an instant payoff.

Nine Books to Thicken This ChapterAI PM psychology reading list: Thinking Fast and Slow, The Design of Everyday Things, Influence, Predictably Irrational, Misbehaving, The Media Equation, Don’t Make Me Think, The Power of Moments, Hooked. Nine books unpack nineteen principles every PM needs, each with an AI product application and a lesson to revisit—pick by the problem you’re stuck on.

Sixteen Effects on One Table, Plus Fourteen Pre-Launch QuestionsAI Product Psychology finale: from perceived performance, labor illusion, trust calibration, and algorithm aversion to honeymoon cliff, cognitive offloading, emotional attachment, pain of paying, anchoring and decoy, and silence bias—a full comparison of sixteen psychological effects and their engineering levers; a checkable fourteen-question pre-launch checklist; eleven themed further-reading picks.

AI Product Psychology · 40 Soul-Searching QuestionsAI Product Psychology interview drill: how will an interviewer, your boss, and a tech colleague test your UX and psychological design? 40 real-scenario questions, each with what they’re assessing, an answer framework, and bonus points: perceived performance, waiting design, peak-end rule, trust calibration, defensiveness, algorithm aversion, AI label discount, emotional attachment, payment and pricing psychology, feedback design.

13

Token Cost Engineering

Understand token costs and reduce waste through denser context and better architecture.

A Business That Bets Against Its UsersAI commercialization is a bet against your users: the better the product, the more they use it, the thinner your margins. Token cost isn't just a finance bill—it's a direct map of latency and throughput. This chapter teaches you how to peel cost down layer by layer.How Tokens Are Counted: BPE and the Hidden “Token Tax”A Token is neither a character nor a word—it's a subword built from statistical patterns. From word-level and character-level extremes to BPE, a quantitative Chinese–English Token comparison, and why Chinese users inherently pay a ~2× hidden Token tax.Price Sheet at a Glance and the Three TiersA full pricing panorama of GLM-4.6, Qwen3-Max, Qwen-Plus, and Qwen-Flash: how T0 Flagship, T1 Workhorse, and T2 High-volume divide the work. Knowing which is expensive isn't enough—watch the price-jump boundaries.

GLM's Short-Output Game: the 200-Token CliffGLM-4.6 tiers by output length: 199 vs 201 output Tokens hit completely different prices—output up 75%, and input gets a retroactive 50% bump too. Drag the slider to watch the bill jump, plus four counter-strategies like task splitting and field tiering.Qwen's Tier Escape: the 32k Red LineQwen tiers by input length: a 33k request bills all 33k Tokens at 2×—not just the overflow. The trap of paying double for RAG junk, plus budget-aware truncation code.Image Tokens: Pixels Pay Tax TooHow do images become Tokens? The core formula (h×w)/token_pixels+2, the 32-pixel alignment tier-jump trap, the resolution curse and diminishing marginal returns. A live resolution→Token calculator and three cost red lines.

Input-Dominated: a 62:1 I/O RatioA chatbot is one question, one answer; an Agent is a loop: every turn's Input includes the full history, so cost balloons as turns pile up. Walk an Excel analysis task turn by turn and watch the I/O Ratio hit 62:1.Four Agent Cost Traps and the Circuit BreakerFour Agent cost traps: tool-return information explosion, the invisible bill of thinking Tokens, infinite loops, and history snowball. Each trap gets a shippable fix: truncation guards, task tiering, forced circuit breakers, and fixed prefix plus summary.

Syntax Layer: Prompts Are Written for MachinesBold ** alone eats 8.5% of Tokens; formatting can be 10%–20% of a prompt. Three rules of thumb: YAML not JSON for complex objects, CSV not JSON arrays for flat lists, force Minified JSON on backend output.Semantic Layer: Double DistillationDon't treat the context window as a dump: attention is O(N²), so long prompts are both expensive and slow—and lost-in-the-middle makes quality worse. Double distillation: dynamic Few-Shot cuts 4000 Tokens to 500; LLMLingua-2 compresses documents 5–20×.Architecture Layer: KV Cache CaveatsKV Cache is the most underrated commercial cost lever: prefix-match cache hits can save up to 90%. Two invisible traps: dynamically switching the tools parameter invalidates the entire cache; a blunt sliding window keeps the prefix forever changing. Chapter cache lifts hit rate from 10% to 80%.Output Layer: Keep the Model's Mouth ShutOutput Tokens cost several times more than input. Three moves to keep the model's mouth shut: explicit negative constraints (No yapping) cut ~30% fluff, polish with Diff instead of rewriting the whole passage, and stop sequences as a hard cut. Save Tokens, save time, and worry less about the model rambling.

Minimalism in the Age of ComputeToken Cost Engineering series finale: saving Tokens is really about raising information density—High SNR = high intelligence. A full recap of pricing traps, the Agent bill, and four-layer optimization, plus further reading across eighteen topics.

14

Data Structures for AI

Use AI examples to understand arrays, caches, search, graphs, queues, and related structures.

It's 2026 — Why Still Learn Data Structures?AI can write code now — so why still learn data structures? Because you don't have to write the code, but you do need to check what AI writes. One metaphor for the whole chapter: a data structure = a way of organizing. Play a round of “find the key” and feel how slow the wrong organizer is.

Arrays: Every Message You Chat Lies in OneA message list is just an array: how numbered cells hold every line you and the AI exchange, why the Context Window “cuts the head, keeps the tail,” and why long chats forget. Then insert into the middle of an array yourself and count how many elements on the right must move — and you'll see why chat history only grows backward.Stacks: The Secret Behind Cmd+Z and Stack OverflowWhat is a stack? Last in, first out (LIFO) is hard to explain in one sentence — play with it and you'll get it: the undo key Cmd+Z's real form is a stack of operation cards, function calls use a stack to remember “where to return,” and recursion without a base case overflows the stack (Stack Overflow). An Agent's subtasks and max-step fuse are the same idea.Queues: An Agent's Work Gets Done in LineWhat is a queue? First in, first out (FIFO) is fairness in line. Run a producer-consumer pipeline yourself: drag request rate and processing speed—see when the queue spins idle and when backlog turns red. Then a 30-second animation contrasts stack vs queue—same A B C in, totally different order out. An Agent's todo list, API rate limits, message queues: their real form is right here.

Hash Tables: Why Lookups Are Unreasonably FastWhy could Set “direct-hit” in lesson one? Drop names into buckets yourself: watch the hash function turn a name into a number, mod 8 for the bucket; see how two names colliding in one bucket get handled; then race “walk the list vs direct hit” on a million rows.Caches: The Invisible Discount on Your AI BillKV Cache and semantic cache are the same move: don't recompute what you've already computed. Watch how prefix tokens “cache-hit, skip compute” across two chat rounds; then drag the hit-rate slider and see a support bot's monthly bill fall from ¥6,000 to a few hundred—the underlying principle behind cost optimization in the Harness core part.

Trees: All a Coding Agent SeesFile trees, JSON, the page DOM, code syntax trees—when AI reads your project, it sees trees. Open a line of code and watch it grow into an AST.Graphs: From Knowledge Graphs to Multi-Agent CollaborationNodes plus relations make a graph: social nets, knowledge graphs, Agent workflow DAGs. Click a node and watch relations ripple out hop by hop.

Vocabulary & Trie: How Tokenizers Cut WordsVectors: RAG Retrieval Is Finding Nearest NeighborsEmbedding turns meaning into coordinates; similarity is distance. Drag the query point on a plane to watch nearest neighbors change, then see why HNSW finds answers instantly among hundreds of millions of vectors.

Recap · Eight Structures, One Decision TableData structures part wrap-up: array/stack/queue/hash table/cache/tree/graph/vector—eight ways of organizing on one decision table, with mottos, strengths, weaknesses, and their real form in AI; then a scenario picker and 6 quick quiz items to check whether you can review AI-written code.

Check AI-Written Code Once ThroughData Structures hands-on: check AI-written code once through. Three tiers—have AI explain the structure it picked (30 min), demand an alternate implementation and compare trade-offs (half a day), pick a structure for your own project (one week). Prompt templates are all copy-ready.

Data Structures · 30 Tough QuestionsData Structures interview practice: how will an interviewer, your boss, and a tech colleague test your data structures? 30 real-scenario questions, each with intent, answer framework, and bonus points: arrays & message list, hash collisions, cache invalidation, trees & AST, graphs & DAG, Trie tokenization, vector retrieval.

15

Algorithms for AI

Build practical intuition for complexity, sorting, divide-and-conquer, search, and sampling.

Big-O: See at a Glance How Long Code Will RunDrag the data-size slider and watch O(1), O(log n), O(n), and O(n²) diverge; 10× the data—who barely flinches, who blows up on the spot. No math required—three interactives build your intuition for how fast code runs.Why Longer Context Costs More: The O(n²) BillAttention makes every token look at every token: drag context length and watch compute and the bill climb with the square—why long chats get slow and expensive. Take last lesson's O(n²) ruler to the LLM itself.

Binary Search: The Optimal Number-Guessing GamePlay a 1-to-100 number-guessing round and feel the range cut in half each guess; a billion items found in 30 tries—how fast log n really is. Binary search's ticket in: sort first.Sorting: Bubble Sort vs Quicksort RaceA head-to-head visual animation of two sorts: watch bubble sort nudge step by step and quicksort leap by partitions; see how lopsided the gap gets once the data grows. Feel the difference between O(n²) and O(n log n) in one race.Sorting's Real Face in AI: RerankPassages RAG retrieves can’t go in as-is: coarse ranking first, then fine ranking. Tweak the weights yourself and watch candidates reshuffle—feeds and search results work the same way. Sorting’s real face in LLM engineering.

Recursion: Break a Big Job into the Same Smaller JobWalk a directory tree, break down tasks, draw a fractal tree—same playbook: call yourself. Watch an Agent recursively split “build a site” into actionable small jobs, then drag a slider and grow a fractal tree from one rule.Divide and Conquer: The Algorithm Behind Context CompressionCompaction in the AI Practicum is really divide and conquer: split a long chat, summarize each chunk, then merge. Run recursive summarization yourself—watch a 3600-token dialogue compress to 200, and see what gets dropped.

BFS and DFS: How an Agent Finds Files in a CodebaseWatch two search personalities in a maze animation: BFS sweeps layer by layer; DFS goes all the way down one path. Coding Agent grep, web crawlers, and mutual-friend recommendations are variants. Cells visited and path length settle it in one run.Greedy and Sampling: Two Personalities When AI Picks WordsAlways picking the biggest bar is greedy decoding; rolling dice by probability is sampling—the algorithms behind Temperature. Same starter, click thrice: greedy is identical every time; sampling differs; higher temperature goes wilder.Beam Search: Look a Few Steps Ahead Before ChoosingOne greedy misstep cascades. Beam Search keeps several candidate paths and explores ahead. On a 4-step word lattice: greedy scores 0.072, Beam=2 finds 0.098, Beam=3 finds 0.101—at roughly 2–3× compute. Feel where “think first, then answer” comes from.

AI Can Solve the Problems—Do You Still Need LeetCode?AI can solve the problems—do you still need LeetCode? Honest talk on interviews in 2026: which roles still test algorithms, how hard, and which stopped caring. Tap your target role to find your seat, then see how interview focus shifted from 2020 to 2026.The Right Way to Learn Algorithms with AIThe right way to learn algorithms with AI: let it set problems, play Socratic examiner, and walk complexity line by line. Three ready-to-copy prompt templates, plus 5 “good approach vs bad approach” tap quizzes—turn AI from doing problems for you into practicing with you.

Summary · Five Algorithm Ideas at a GlanceAlgorithms Part Summary: complexity, search & sorting, recursion & divide-and-conquer, graph search, greedy & sampling—five ideas on one comparison table, each mapped to a real mechanism in AI. Then 8 scenario-spotting questions to test whether you can spot at a glance “which idea this problem needs.”

Run a Complexity Checkup on AI-Written CodeAlgorithms Part hands-on: run a complexity checkup on AI-written code. Three tiers—have AI self-report complexity and point out the bottleneck line (30 min), demand one-tier optimization and spell out the cost (half a day), write a benchmark to verify it isn’t bluffing (one week). Prompt templates are all copy-ready, plus a checkup report generator.

Algorithms · 30 Must-Answer QuestionsAlgorithms Part interview drill: how will an interviewer, your boss, and a tech colleague test your algorithms? 30 real-scenario questions, each with what they’re assessing, an answer framework, and bonus points: Big-O intuition, attention O(n²), binary-search premises, the Rerank funnel, recursion risks, BFS vs DFS, greedy sampling, and whether you still grind problems in the AI era.

16

Grok Build Deep Dive

Read Grok Build as an optional source deep dive into a production coding agent.

How 79 Workspace Members Form a ProductReconstructing the Grok Build Cargo workspace map from real source code, explaining the 79 members and the composition entry, TUI, Agent host, state, and inference layers.Why Rust Makes Sense: Separating Facts from InferencesDistinguish verifiable Rust source-code facts from course inferences in Grok Build using real Cargo configurations, with a public behavior comparison.From Real main() to First Sampling RoundTrace the real entry xai-grok-pager-bin/src/main.rs through headless, leader, stdio, and TUI modes to SessionActor and SamplerActor.

Session Actor: Thread, State & Cancellation BoundariesAccurately understand Grok Build's per-Session OS thread, Tokio LocalSet, SessionActor turn loop, ChatStateActor, and the cancellation mechanism.Compaction: 85% Threshold and Optional Two-PassVerify CompactionPolicy's real fields, default values, 85% threshold, optional two-pass, and token estimation logic.PromptContext: Inspectable Rendering InputVerify PromptContext real fields, Serialize/Deserialize derive, three TemplateOverride variants, and the ToolBridge rendering pipeline.

Process-Level External Toolset Preset RegistryA source-code walkthrough of the process-level external Toolset preset registry in config.rs, covering visibility and registration timing.ToolKind Provides Default Read-Only SemanticsBased on tool_taxonomy.rs, explains real ToolKind branches, is_read_only default classification, and permission decision boundaries.Implementation Families, Registry, and Dynamic MCPA hands-on guide to tool implementation families and dynamic MCP calls, based on the real xai-grok-tools codebase, ToolBridge, and registry.Canonical Input Is a Stable ProjectionExplains Grok Build canonical fields, CanonicalToolMeta v1, and canonical input projection.

Estimation, Percentage, and Strict ThresholdsExplains usage_percentage, threshold boundaries, and headroom based on the real xai-token-estimation functions.From File Changes to Hybrid RankingBased on memory/search.rs: explaining FTS5, sqlite-vec, time decay, source weight, MMR, and sync fallback.The Real Mechanism of DreamUnderstand Dream's gating, idempotency, locking, and failure recovery from the real Grok Build source code.

How AgentDefinition and Persona MergeA source-code walkthrough of AgentDefinition, Persona, and EffectiveRuntimeConfig based on real Grok Build code.Four Isolation Dimensions of Sub-AgentsUnderstanding the true semantics of ContextSource, ResumeSourceData, SubagentIsolationMode, and worktree.How to Organize Multi-Agent SystemsUnderstand multi-Agent organization through Grok Build's subagent_coordinator, AgentDefinition, and Persona constructs.

Five Sandbox ProfilesAccurately explains Grok Build's five sandbox profiles: workspace, devbox, read-only, strict, and off.From Tool Request to Restricted Execution: The Complete Authorization ChainA walkthrough of hooks, permission resolution, bash segmentation, user confirmation, and sandbox execution based on real Grok Build source code.Hooks: Explicit Deny to BlockGrok Build Source Course Lesson 19: Hook events, matchers, blocking semantics, and fail-open boundaries.

MCP: Connection Is Just the BeginningGrok Build Source Course Lesson 20: MCP client, OAuth, tool visibility, and connection recovery Marketplace： Discovery, Installation, and Execution LayersGrok Build Source Course, Lesson 21: Plugin Marketplace, plugin discovery, and trust boundaries

Full Comparison: Calibrate Evidence Before Trade-offsComplete side-by-side comparison of Grok Build and Claude Code: layered analysis with source evidence and public product behavior.Engineering Retrospective: Capabilities and Boundaries, Read TogetherGrok Build Source Course Lesson 23: Engineering retrospective and applicable boundaries based on source code and README.Coding Agent Design WorkbenchCoding Agent Design Workbench: nine system dimensions and submittable capstone deliverables

Dissecting Grok Build · 30 Soul-Searching QuestionsChapter 6 real-world exam prep: how will interviewers, your boss, and technical colleagues test you on production-grade Coding Agents? 30 real-scenario questions, each with the examiner's intent, an answer framework, and bonus points: runtime loop, context compaction, tool classification, memory retrieval, security sandbox, MCP integration.

17

DeepSeek Harness Deep Dive

Read DeepSeek Harness as an optional source deep dive into a plugin-first agent runtime.

Everything Is a Plugin: Announcement vs SourceWhich files in the repo match the four modes and plugin ecosystem from the launch post — this lesson maps them one by one Profile / Bundle / Patch: How Far Users Can Reshape the ProductDSH’s three-layer config: Bundle as release defaults, Profile and Home as user overlays, --patch for one-shot experiments, and the trade-offs of whole-entry replace.

Model-visible ⟺ logged: An Invariant That Crashes in Your FaceEverything the model sees must be reconstructable from the log: the two design ideas behind this DSH invariant—single source of truth, fail the check then crash followup / steer / inject: Dual-Queue InboxA queue metaphor for the next-turn and next-step queues: how followup, steer, and inject differ, why claim is atomic, and why post-abort input is force-downgraded What Happens After Esc: Cancel, Crash Recovery, and Re-entryDSH cancel and crash recovery: one AbortSignal per turn, write turn/end even on interrupt, synthesize interrupted closers on cold load, and keep every persisted result after a crash restart.Goal: Provenance Is PermissionDSH's Goal subsystem lets an Agent run dozens of turns on its own without letting it extend its own life: permission to change the goal follows message provenance; authority.ts authenticates with host-stamped source metadata—Prompt Injection defense at the mechanism layer.

Compaction Dual Paths & replaceGenerationTwo design ideas behind DSH auto-compaction: pack proactively when nearly full, recover after a blow-up — two triggers, each owning a segment; before overflow retry you must check the replaceGeneration number, and if packing didn’t work, no retry — so you don’t burn money in a death loop.Token Accounting: Replay for Decisions, Projection for DisplayDSH splits token accounting in two: compaction decisions use replay-style measure(); the UI shows projectedTokens projection. The two numbers may differ—and that's intentional. Contrast Grok Build's single bytes/4 source and Claude Code's progress counters.Spill: When Tool Output Is Too LargeBased on real dsh-spill-policy source: when plain-text tool results exceed maxInlineBytes, DSH spills the full text to disk, leaves only head/tail previews plus a retrieval credential in context, and lets the model recover with read/grep anytime — contrasted with Claude Code and Grok Build truncation.Session Search & Cross-Session CitationsBased on real dsh-session-query and dsh-session-reference source: DSH session search with FTS, current/shadowed/log-only three-state labels, frozen snapshots and untrusted marks for cross-session citations — contrasted with Claude Code extractive memory and Grok Build hybrid retrieval.

Tool Execution Pipeline: Three-Stage Cascade & Monotone GuardDissect DeepSeek Harness's tool execution pipeline: what pre-execute, execute, and post-execute each own; why monotone Guard has no allow option in the type system; and how a denied call materializes as a model-visible error result.Tool Output Contract: Value Separated from DisplayDissect DeepSeek Harness’s tool output contract: execute returns a schema-validated canonical value, render projects model content, presentResult projects a UI card. Same result — what the model sees and what people see can differ; persistence stores content, not value.File-Edit Engineering: Read Before WriteDissect DeepSeek Harness’s file-edit defenses: the read/edit/write trio plus the fs-observation-policy observation ledger — unread files can’t be edited; after a read, external changes raise FS_STALE_VERSION; unique literal matches stop wrong-spot edits.DSH-Only Tool Surface: terminal / lsp / jobsWhat the tools others lack each solve: the terminal six-pack keeps REPL state, lsp closes four semantic query types, jobs fold background bash, PTY, and subagent into one task list.

Approvals & Permissions: Two Knobs, One DropdownDSH splits permissions into two independent knobs — sandbox mode and approval policy; presets are just shortcuts for common combos. From real user-approval and permission-presets source: fail-closed approval and derived custom.Sandbox: From seatbelt to the Execution WorldWhere DSH draws the sandbox boundary: ctx.sandbox only covers same-kernel child processes, with policy carried per call; ctx.fs and ctx.subprocess share a path namespace, so swapping in an E2B adapter moves the whole remote Execution World.

Code Mode: One Program Beats Many Tool RoundsAnnouncement calls it PTC, source calls it code mode: the model writes one program, the sandbox runs many tool calls in one go — five round trips become one

Subagent Is a Seam: In-Process to Delegating Claude CodeDSH makes sub-Agents a capability seam: spawn, fork, Claude Code, and Codex are all providers in the ctx.subagents registry — interface stays, impl swaps, insufficient capability fails loud.workflow / schedule / plan / todo: Orchestration PrimitivesDSH splits multi-step orchestration into four independent primitives: workflow lets the model write scripts, schedule handles in-session timers, plan is a logged collaboration stance, and todo is a full-replace progress list. Why they were never merged into one monolith.Skill, Preset, and Self-ModificationDSH’s four official modes are just four YAML files; cordis_define and cordis_run let an Agent grow new tools for itself at runtime. How the trust boundary is drawn, and how the layered skill registry shadows.

MCP & Extensions: Two Paths to External ToolsDSH’s two paths to external capability: MCP bridges tool servers in the ecosystem; Extensions let the model write and run Cordis plugins live in the harness. Naming with hash anti-collision, disconnect generation rollback, and bridging only tools — one page covers the trade-offs.LLM Adapter Layer: Single Attempt, Explicit Retry, Dual-Stream PersistenceDSH’s LLM adapter layer: one adapter call is one provider attempt, retries are explicit events in the persistent log, and the raw chunk stream is stored separately from the derived message stream. Walked through real assembler.ts and agent.ts source.Testing a Nondeterministic SystemAgent behavior depends on model output — how do you get reproducible tests? DSH’s answer: deterministic replay turns logs into test assets, property-based testing sweeps interleaved sequences, and a fault server built to trick LLM clients rehearses every way a connection can die.

Persistence Governance: Versions, fork Boundaries, Refuse-to-ParseHow session-log formats evolve: a monotone integer version, direction-aware read rules, refuse-to-parse for unknown events by default, and fork’s dual boundaries Credentials, Settings, Storage, and TelemetryCredentials are resolved per operation so rotation needs no restart; settings files use a cross-process lock against torn writes; storage schema refuses migration; one anonymous id serves three consumers Multi-Entry & Typert: One Kernel, Five FacesWeb, headless, ACP, Python SDK, and HTTP API — five entries share one DSH kernel: an entry is only a protocol-translation skin; Typert keeps one type change in sync across every client

Agent Notes & AGENTS.md: Discipline for Building AI with AIDSH uses four-state design notes and a hard AGENTS.md contract written for AI, stopping decision amnesia and rotting docs in repos where AI writes code at scale — ready for teams to copy right away.KV Cache Is an InterfaceDSH treats system-prompt prefix stability as a compatibility promise: 215 package docs carry a fixed KV Cache effect section, tool order is normalized by a central list, and interpolation format errors throw instead of shipping a bad prompt.

Finale: Five Engineering Views — What Should We Copy?A master table of design philosophy across five harnesses — DSH, Claude Code, Grok Build, Codex, OpenCode — plus a minimum copy list and a scale-trap list; build your own architecture list with the design-decision buffet.

19

Open Source and Local Models

Understand weights, licenses, distillation, and running models locally.

What Are Weights? Everything a Model KnowsWhat are weights? They are the pile of floating-point numbers left behind once training ends, and everything a model can do lives in them. Get the weight file and you can run exactly the same model on your own machine. Understand this and every later argument about open source makes sense.Real vs. Fake Open Source: How to Read a LicenseEvery vendor claims to be open source, and they mean very different things. Three questions settle it: can you download the weights, can you use them commercially, and can you train new models on the output? The same yardstick applied to Qwen, Llama, Mistral, DeepSeek, and API-only models.Open Source Is a Business: What Each Vendor Is AfterOpen-sourcing costs money, released weights can never be taken back, and no vendor is doing charity. The open-source strategies and revenue paths of Meta, Google, Mistral, Alibaba, DeepSeek, and OpenAI, plus why the number of derivative models is the key measure of ecosystem influence.

Emergence: Why Abilities Show Up All at OnceModel capability does not grow smoothly with parameter count. Some abilities score near zero on small models and then appear once scale crosses a certain order of magnitude — a phenomenon called emergence. This page covers six common cases, the academic dispute over whether emergence is real, and what it means in practice for model selection.Why Make Models Smaller: The Motivation Behind DistillationIf bigger models are more capable, why is the industry working so hard to make them smaller? Cost, latency, and data compliance are three real-world constraints that rule flagship models out of actual projects. Distillation is how a large model teaches a small one, saving the cost of training from scratch.How Distillation Works: From Teacher to StudentThe full knowledge distillation pipeline: assemble a question set, have the teacher answer while keeping its reasoning chain, replace right/wrong labels with probability distributions, train the student, then evaluate and iterate. Uses the six distilled models DeepSeek open-sourced alongside R1 to show what real distillation output looks like.The Cost of Distillation: Models Are Getting More AlikeDistillation gives small models capability quickly, and the price is that the student inherits everything about the teacher: verbal tics, formatting habits, biases, and even its sense of identity. When large numbers of models are built on a handful of bases, output diversity drops sharply and the premise behind multi-model cross-validation may not hold.

How Large a Model Can Your Computer RunPick your GPU or Mac model and see in real time which open-source models you can run. Includes the VRAM estimation formula, an FP16 versus INT4 quantization comparison, the difference between NVIDIA dedicated VRAM and Apple unified memory, and the mismatch between MoE memory footprint and speed.Getting Started with Ollama and LM StudioRun open-source models locally with Ollama or LM Studio. Includes the full set of commands from install to running, how to read a model tag, how to choose between Q4 and Q5 quantization, and the most common traps such as a context length that blows out your VRAM.

20

Self Tests

Use self tests to check weak spots and decide what to revisit.

7 Chapter Quizzes · 350 QuestionsAI Course Quiz Center: seven chapter quizzes with 350 technical questions, plus a full-site comprehensive exam spanning all chapters. Covers LLM Fundamentals, AI Harness, Product Engineering, Design Patterns, Self-Improvement, Coding Agent internals, and Vibe Coding. Two modes: mock exam and sequential drill, with wrong-answer explanations.

LLM Fundamentals · Chapter Self-AssessmentChapter 1 Self-Assessment: 50 questions covering the three stages of LLM training, Token mechanics, hallucination types, and four mitigation strategies. Mock exam + sequential drill modes, with explanations.AI Harness · Chapter QuizChapter 2 Quiz: 50 questions covering context engineering, advanced Prompt techniques, injection attack and defense, Agent tool calls, and KV Cache cost optimization. Simulated exam + sequential drill modes.

From Demo to Product · Chapter QuizChapter 3 Quiz: 50 questions covering image-generation productization, Agent loop control, context compression, long-term memory, multi-Agent collaboration, and the MCP ecosystem. Mock exam + sequential drill modes.AI Engineering Design Patterns · Chapter QuizChapter 4 Quiz: 50 questions covering Workflow patterns, Context Engineering, ACI tool design, Agent evaluation, and secure containerization. Mock exam + sequential drill modes.

Harness & Self-Improvement · Chapter QuizChapter 5 Quiz: 50 questions covering Harness design patterns, context engineering auto-evolution, workflow search, recursive self-improvement, and risks. Mock exam + sequential drill modes.Vibe Coding Methodology · Chapter QuizChapter 7 Quiz: 50 questions covering Vibe Coding philosophy, workflow control, quality baseline, documentation practices, and safety gates. Mock exam + sequential drill modes.Dissecting Grok Build · Chapter QuizChapter 6 Quiz: 50 questions covering Coding Agent runtime, Compaction, tool approval, memory retrieval, five sandbox profiles & MCP integration. Mock exam + sequential practice modes.

Full-Site Comprehensive Exam · 35 Questions from 7 ChaptersFull-Site Comprehensive Exam: 35 questions drawn evenly from 350 across seven chapters, covering LLM Fundamentals, AI Harness, Product Engineering, Design Patterns, Self-Improvement, Coding Agent internals, and Vibe Coding. Scored on a 100-point scale; wrong answers link back to the exact lesson.

21

One-Person Company Basics

Learn the ownership and compliance basics around brands, code, domains, entities, and equity.

Six Weapons: The Right Lock for Each AssetCode, names, algorithms, articles, and URLs are five different things in law—each needs its own lock. Patents, trademarks, software copyright, copyright, trade secrets, and domain names: what each weapon protects, what it costs, and when to use which—cleared up in one lesson.Clear Rights Early: Early Is Investment, Late Is LossThe same rights-clearance move is an investment before launch and a loss after someone beats you to it. A dual timeline contrasts early clearance with late salvage, plus four things to finish before you write the first line of code.

One Trademark Isn't Enough: Take Classes 35, 42, and 41 TogetherOne trademark registration won’t cover a business. Class 35 is the ticket to promotion and monetization, Class 42 covers software and SaaS, Class 41 covers courses and content—and Apps also need Class 9. Includes an interactive Class Checklist Generator and the specimen-consistency trap.After a Refusal: Clear the Blocking MarkA trademark refusal doesn’t mean the name is gone. A real case we walked end to end: refusal, three-year non-use cancellation, and review of refusal—who bears the burden of proof, and what the 2027 Trademark Law renumbers.Lock Down the Domains: How a ¥100,000 Ask Became ¥5,000A domain listed by a broker at ¥100,000 closed at about ¥5,000. Why? Because we already held the core extensions—their leverage went to zero. A real negotiation we finished ourselves: the strategic value of full-suffix registration.

Software Copyright and Recordation: Rights Arise Automatically, Evidence You EarnCopyright arises automatically when a work is created—so why pay to register? Evidence. How software copyright registration, copyright recordation, and trusted timestamps each work; who owns copyright when outsourcing without an ownership clause; plus a “who owns the copyright” drill.Patent or Trade Secret: File First, Disclose LaterPatents trade disclosure for protection; trade secrets trade secrecy for protection. A decision tree shows how to choose. Worse: if you ship before filing, novelty is gone forever. Includes an interactive decision tree and a novelty self-check.

ICP Filing Is the Foundation; an ICP License Is the Right to ChargeICP filing and an ICP license are two different things: filing is the prerequisite for a site to be reachable normally in China—everyone needs it; an ICP license is the right to charge users, starting at RMB 1,000,000 registered capital. Get business scope wrong on incorporation day and commercialization dies on a company-registry change.Don't Inflate Registered Capital: You Must Pay It In Within Five YearsUnder the Company Law effective July 2024, subscribed registered capital must be paid in within five years of incorporation. How much you write is set by the business model—not by face. Plus the two traps solo founders hit hardest: commingling personal and company money, and gray-market payment collection.

Partners Must Complement: Don't Partner with Someone Whose Skills Overlap YoursShould a one-person company bring on a partner? Two rules: complementary skills beat shared passion—no skill overlap; radical trust—if you can't unconditionally believe they're stronger than you in their domain, don't partner. Includes a partnership decision quiz and a trial-period playbook.Equal Thirds Is the Most Expensive Kind of FairnessEqual thirds look fair; they're a decision deadlock: amending articles of association, changing capital, merger, division, or dissolution needs two-thirds or more of voting rights—any two people can't lock it in alone. Why the core founder should hold 70%+, with an equity structure simulator.Don't Split Equity by Gut Feel: Run Five DimensionsFive dimensions of equity: overall capability 30%+, capital contribution 15–20%, opportunity cost ~20%, startup idea source 10–15%, ownership of responsibility ~20%. Opportunity cost and responsibility are easiest to miss. Includes a five-dimension scorer for 2–5 people—see who clears half and who clears two-thirds.Sign a Nominee Agreement First; File the Company Registry Change When It VestsThree early-stage equity disciplines: skip complex holding platforms and option pools, sign a nominee agreement before any company registration change, and write exit and repurchase terms up front. The legal force of nominee agreements and their limits—and why a departing co-founder’s refusal to sign can lock up the company.

Server Location Doesn't Matter—Where Your Users Are DoesPutting servers overseas and wrapping an offshore model API does not dodge filing duties—that's fake going-global. The Interim Measures for the Management of Generative Artificial Intelligence Services apply based on serving the public inside China, not server location. Real going-global means a compliant entity, local law, and real local-user operations.The Long Tail Big Tech Ignores Is Where a Solo Founder WinsBig tech has an ROI bar—long-tail niches that don’t clear it are a one-person company’s opportunity. How to choose between digging into domestic long tail and doing overseas properly. Plus China domestic model picking: capability gaps are already small, price gaps still huge—how to choose by scenario.The New Startup Formula: Find Buyers First, Then Build with AIOne-Person Company (OPC) finale: the new startup formula replaces the old one—find a niche market with paying buyers first, validate fast, then build the product with AI. Includes a 15-lesson panorama, a 15-item compliance checklist, the seven-character mantra and cutting losses, plus three things you can do this week.

22

SEO and GEO

Understand how search engines and answer engines discover and cite AI products.

You Built It. Why Isn't Anyone Coming?AI drove the cost of shipping down to almost nothing. The doors to being found got more crowded. Three traffic doors, three temperaments: search engines when people look, AI engines when people ask, social when someone else talks about you. Run the indexing funnel and see which layer your product is stuck on.

The SEO Minimum Viable Checklist: Crawlable, Readable, IndexableThe SEO minimum viable checklist is three things: crawlable (don't render the body in JS alone), readable (one topic per page, a human title), indexable (sitemap, robots, webmaster-tool submit). Flip the crawl simulator yourself and see what the crawler actually got.GEO: Get AI Engines to Cite YouUsers no longer just search — they ask. GEO's four levers: self-contained paragraphs (one-sentence answer blocks), FAQ structured data, llms.txt, and freshness signals. A citation-auction demo shows which passages AI prefers to cite, plus an AI crawler roster and a ready-to-use FAQPage generator.

Dissecting This Site: Google Indexes 693 Pages, Bing Only 50Xiaoshan Academy's real SEO/GEO audit in August 2026: what it means that Google indexes about 693 pages while Bing has only about 50, why the sitemap missed every English and Korean page, how Q&A pages marked only as Article threw away FAQ citations, and how dropping bot logs made GEO unmeasurable. For each gap: how we found it, what we changed, how we check.

Solo Priorities: Write Questions People Actually SearchA one-person company has no SEO team — only the ability to rank its bets. The method: write the question users actually type, then add features; one page, one question; trade real experience for trust. This lesson also covers the anti-list: bought backlinks, keyword stuffing, AI mass-spam pages — which tactics will get you killed.Getting-Found Launch Checklist: One Last Pass Before You ShipGetting Found finale: a 12-item checkable SEO/GEO launch list — body visible, sitemap, canonical, one-sentence answer block, FAQ structured data, llms.txt, crawler logs. Plus four measurement definitions for 'how do I know it worked,' paired with the Taste and Interaction lists as one pre-launch set.

No matching lessons

Bookmark LearnAI0Use Cmd+D or Ctrl+D to resume later

Source AGPL-3.0

Loading lesson...

The lesson could not be loaded

Reload it or choose another lesson from the course contents.

←Previous lessonThree Grader Types: Code, Model, HumanNext lessonWhy Agents Fail at Long Tasks→