The Spark

Chapter 1Start with Hardware (or Don't)

The first question everyone asks: do I need my own GPU to work with AI?

The honest answer: no. Not at first. Cloud APIs like Claude, GPT, and Gemini are the fastest way to start. You sign up, get an API key, and you're productive in minutes. For many use cases — writing, analysis, coding assistance — cloud AI is all you'll ever need.

But there's a ceiling. Cloud AI charges per token. Every question you ask, every response you get, costs money. When you're experimenting — trying things, failing, learning — those costs add up. More importantly, your data leaves your network. Every prompt, every document you upload, every database query you ask about — it crosses the internet to someone else's server.

Local hardware removes both limits. Once you own the GPU, inference is free. Run a million tokens at 3 AM because you're curious about something. Upload your entire codebase without worrying about confidentiality. Experiment recklessly. That freedom changes how you think about AI — it stops being a service you pay for and becomes a tool you own.

What You Actually Need

If you're just starting (budget: $0): Use cloud APIs. Claude's free tier, ChatGPT's free tier, Google's Gemini. Get comfortable with what AI can and can't do before investing in hardware. This stage is about learning the language — prompting, context management, understanding model behavior.

If you're ready to go local (budget: $600–2,500): A used NVIDIA RTX 3090 (24GB VRAM) runs 7B–13B parameter models comfortably. That's enough for a surprisingly capable assistant. An Apple Mac with M-series chip works too — the unified memory architecture means even a MacBook Air can run small models. At the higher end, an RTX 4090 gives you more headroom for larger models.

If you're building infrastructure (budget: $5,000+): Multi-GPU setups, dedicated inference servers, or machines like the NVIDIA DGX Spark. This is where you run 30B+ parameter models at production speed, serve multiple users simultaneously, and start thinking about your AI stack as infrastructure rather than a tool.

We started at the high end — two DGX Spark machines with GB10 Blackwell chips, connected by 200GbE InfiniBand. That's not typical, and it's not necessary. But it taught us something important: the hardware is just the beginning. The real work is what you build on top of it.

Chapter 2Your First Local Models

The moment you run a language model on your own machine for the first time, something clicks. It's not magic anymore — it's software. Software that runs on your hardware, processes your data, and produces output without calling home.

Ollama is the simplest path to that moment. It's a tool that downloads and runs language models with a single command. No configuration files. No dependency hell. No PhD required.

What happens when you run that command? Ollama downloads a 4–5GB model file, loads it into your GPU memory (or CPU RAM if no GPU), and starts a conversation. The model runs entirely on your machine. No internet required after the initial download. No API key. No token counter ticking up.

Choosing Your First Model

There are hundreds of models available. That's overwhelming. Here's how to think about it:

Model size matters. Models are measured in "parameters" — 7B means 7 billion. More parameters generally means more capable, but also needs more RAM. A 7B model runs on almost anything. A 13B model needs a decent GPU. A 30B+ model needs serious hardware. Start with 7B.

Different models have different strengths. Qwen is excellent for general conversation and handles multiple languages well. DeepSeek Coder is optimized for programming. Phi is designed to be small and fast — good for older hardware. Llama is Meta's general-purpose family. Try several. They're free to download.

The model you use daily will surprise you. Most people expect to need the biggest, most powerful model. In practice, a well-prompted 7B model handles 80% of daily tasks. You only need the heavy models for complex reasoning, long documents, or specialized domains.

What You'll Notice

Local models are slower than cloud APIs. A cloud model like Claude responds in seconds because it runs on massive server clusters. Your local model might take 10–30 seconds for a long response. That's normal. The tradeoff is privacy, cost, and the freedom to experiment without limits.

Local models are also less capable than the largest cloud models. Claude Opus or GPT-4 have hundreds of billions of parameters and months of fine-tuning by large teams. Your 7B local model is a different class of tool. Think of it as the difference between a company car and a taxi — the taxi is nicer, but the company car is always available and doesn't charge per mile.

Chapter 3Inference Engines: Ollama vs vLLM vs TensorRT-LLM

You've got a model downloaded. Now you need software to actually run it — to feed it prompts and get responses. This software is called an inference engine, and your choice here matters more than most people realize. It affects speed, memory usage, how many users can share the model, and how much of your GPU you're actually using.

There are three major options, each built for a different stage of the journey.

Ollama — The Easy Button

Ollama is where most people start, and for good reason. It's the simplest way to run models locally. One command to install, one command to run a model, and it handles everything behind the scenes — downloading model files, managing GPU memory, providing an API endpoint.

How it works: Ollama wraps llama.cpp, a C++ inference engine optimized for running on consumer hardware. It adds a friendly CLI, automatic model management, and an OpenAI-compatible API so other tools can connect to it. When you run ollama run qwen2.5:7b, it downloads the quantized model, loads it onto your GPU (or CPU), and starts serving requests.

Best for: Personal use. Experimentation. Getting started. Running models on a laptop or single-GPU workstation. The "I just want to talk to an AI locally" use case.

vLLM — The Production Server

When you outgrow Ollama — when you need to serve multiple users, maximize GPU throughput, or run models at scale — vLLM is the next step. It's an open-source inference engine built specifically for high-throughput, low-latency serving.

How it works: vLLM introduces two key innovations. First, PagedAttention — a memory management technique that eliminates wasted GPU memory by paging attention key-value caches, similar to how operating systems page virtual memory. This means you can serve more concurrent requests with the same GPU. Second, continuous batching — instead of waiting for one request to finish before starting the next, vLLM processes multiple requests simultaneously, keeping the GPU busy at all times.

Best for: Team deployments. API endpoints serving multiple clients. Scenarios where throughput and latency matter. The "I need to serve 10+ concurrent users from my own hardware" use case.

TensorRT-LLM — Maximum Performance

If vLLM is the production server, TensorRT-LLM (TRT-LLM) is the race car. Built by NVIDIA specifically for their GPUs, it squeezes every last drop of performance out of the hardware through deep optimization.

How it works: TRT-LLM compiles your model into an optimized "engine" — a GPU-specific binary that uses NVIDIA's TensorRT framework. This compilation step analyzes the model architecture and your specific GPU, then generates optimized CUDA kernels, fuses operations together, and applies hardware-specific tricks like FP8 quantization on Hopper/Blackwell GPUs. The result is significantly faster inference than generic frameworks.

Best for: Dedicated NVIDIA GPU servers where maximum performance matters. Production deployments with latency SLAs. The "every millisecond counts and I have NVIDIA hardware" use case.

Which One Should You Use?

Chapter 4Hugging Face: The Library of AI

If you've spent any time in the AI space, you've seen the name Hugging Face 🤗. Think of it as the GitHub of machine learning — an open platform where researchers and companies publish models, datasets, and tools. Understanding Hugging Face is essential because almost every model you'll ever run came from there, or at least passed through it.

What Hugging Face Actually Is

The Hub: Hugging Face Hub hosts over 500,000 models. When someone trains a new model — whether it's Meta releasing Llama, Alibaba releasing Qwen, or a university researcher releasing a specialized medical model — they publish it on Hugging Face. Each model has a page with documentation, benchmarks, example code, and download links. It's like a package registry, but for AI models.

The Transformers Library: Hugging Face also maintains transformers, the most widely used Python library for working with AI models. It provides a unified API for loading, running, and fine-tuning thousands of different models. Instead of learning a different codebase for each model architecture, you learn one library and it handles the rest.

Datasets: The Hub also hosts datasets — collections of text, images, audio, and other data used to train and evaluate models. If you're fine-tuning a model (more on that later), you'll likely find a relevant dataset on Hugging Face.

How You'll Use It

Finding models. Go to huggingface.co/models. Filter by task (text generation, translation, summarization), size, language, or license. Read the model card — it tells you what the model was trained on, how it performs, and any known limitations. This is where you discover models you didn't know existed.

Downloading models. Every inference engine (Ollama, vLLM, TRT-LLM) ultimately gets its models from Hugging Face. Ollama repackages popular models for easy download. vLLM loads directly from Hugging Face model IDs. Even when you use another tool, the model file originated on the Hub.

Model cards. Every good model on Hugging Face has a model card — a README that describes the model's capabilities, training data, benchmarks, and limitations. Always read the model card. It tells you if the model was trained on code (useful for programming tasks), what languages it supports, whether it handles tool use, and what context length it supports. A 5-minute read can save you hours of frustration using the wrong model for your task.

Spaces. Hugging Face Spaces are hosted demos of models and applications. Before downloading a 10GB model, you can often try it in the browser first. Someone has probably built a demo. Search for it.

Understanding Model Formats

Models on Hugging Face come in different formats, and this matters:

PyTorch (.bin / .safetensors): The standard format. Full-precision weights. This is what the model was originally trained in. Large files, but most tools can load them directly.

GGUF: The format Ollama and llama.cpp use. Pre-quantized (compressed) for efficient inference. Smaller files, faster loading, runs on consumer hardware. Look for GGUF versions of models if you're using Ollama.

GPTQ / AWQ: Quantized formats optimized for GPU inference. Used by vLLM and other GPU-focused engines. Good balance between size and quality.

Safetensors: A secure, fast alternative to PyTorch's pickle format. Increasingly the default. Loads faster and can't contain malicious code (unlike pickle files).

Chapter 5Quantization: Making Big Models Fit Small Hardware

Here's a fundamental problem: the best models are too big for most hardware. A 70B parameter model in full precision needs about 140GB of memory. That's more than any single consumer GPU. Even a 30B model needs ~60GB — more than most GPUs have.

Quantization is the solution. It's a technique that reduces the precision of a model's numbers — storing weights in fewer bits — to shrink the model size and speed up inference. It's the reason you can run a 70B model on a single 24GB GPU. And understanding it is the key to getting the most out of your hardware.

How Quantization Works

A neural network is, at its core, billions of numbers (called "weights"). In full precision, each weight is stored as a 16-bit floating-point number (FP16). That's 2 bytes per weight. A 7B model × 2 bytes = 14GB. A 70B model × 2 bytes = 140GB.

Quantization reduces the precision of these numbers. Instead of 16 bits per weight, you can use 8 bits (INT8), 4 bits (INT4), or even fewer. The math is straightforward:

A 70B model that needs 140GB at FP16 only needs ~35GB at 4-bit quantization. Suddenly it fits on a single high-end GPU, or across two consumer GPUs.

The Quality Tradeoff

Less precision means less accuracy. Every time you quantize, you lose some information. The question is: how much quality do you lose, and does it matter for your use case?

8-bit (INT8/Q8): Nearly indistinguishable from full precision. Most benchmarks show less than 1% degradation. This is the "safe" quantization — you lose almost nothing. If your hardware can handle the size, use this.

4-bit (INT4/Q4_K_M): The sweet spot for most people. Quality degrades slightly — maybe 2–5% on benchmarks — but the model is 4× smaller. For most conversational tasks, coding assistance, and general use, you won't notice the difference. This is what Ollama uses by default for most models.

3-bit and below: Noticeable quality loss. The model starts making more mistakes, especially on complex reasoning tasks. Useful for getting a very large model onto small hardware, but expect compromises. Think of it as running a blurry photo — you can see what's in it, but you miss details.

Quantization Methods

Not all quantization is created equal. Different methods handle the precision reduction differently:

GGUF quantization (for Ollama/llama.cpp): Uses mixed precision — more important layers keep higher precision while less important ones get compressed further. Variants like Q4_K_M (4-bit medium), Q5_K_S (5-bit small), Q6_K (6-bit) give you fine-grained control over the size/quality tradeoff. This is what you'll use most often.

GPTQ: A post-training quantization method that calibrates the quantization using a small dataset. Produces high-quality 4-bit models optimized for GPU inference. Works well with vLLM.

AWQ (Activation-Aware Weight Quantization): Similar to GPTQ but focuses on preserving the weights that matter most for activation patterns. Often produces slightly better quality than GPTQ at the same bit width. Also works with vLLM.

FP8 (8-bit floating point): A newer format supported by NVIDIA Hopper and Blackwell GPUs. Unlike integer quantization, FP8 keeps the floating-point format, which preserves more information. Hardware-accelerated on supported GPUs, making it very fast. TRT-LLM excels here.

Chapter 6NIM Containers: Production-Ready AI in a Box

Setting up inference engines manually works, but it's tedious. You need the right CUDA version, the right Python packages, the right model format, and the right configuration. One wrong version and everything breaks. NVIDIA NIM (NVIDIA Inference Microservices) solves this by packaging everything into a Docker container that just works.

What NIM Is

A NIM container is a pre-built, optimized Docker image that contains:

— The model, pre-converted to the optimal format for your GPU
— The inference engine (usually TensorRT-LLM), pre-compiled
— An OpenAI-compatible API endpoint
— Health checks, metrics, and monitoring endpoints
— All dependencies, CUDA libraries, and drivers

You pull the container, run it, and you have a production-quality inference endpoint. No compilation. No dependency management. No "it works on my machine" problems.

Why NIM Matters

Optimized for your hardware. NIM containers detect your GPU type and load the optimal engine. If you have an H100, it uses FP8 with Transformer Engine. If you have an A100, it uses INT8 with different optimizations. If you have a consumer GPU, it adjusts accordingly. You get near-maximum performance without manual tuning.

Consistent deployment. The container that works in your test environment works the same way in production. No "but I had CUDA 12.1 locally and production has 12.4" surprises. Everything is pinned and tested together.

Enterprise features. NIM containers include health checks (/health), metrics (/metrics in Prometheus format), and structured logging. These aren't nice-to-haves — they're essential for running inference as a service.

Model catalog. NVIDIA maintains a catalog of NIM-ready models at build.nvidia.com. Major models from Meta, Mistral, Alibaba, and others are available. New releases are typically NIM-ready within days.

When to Use NIM vs Raw Engines

Use NIM when: You want production deployment with minimal effort. You're running NVIDIA GPUs. You need monitoring and health checks. You're deploying to multiple machines and need consistency. You value stability over bleeding-edge model support.

Use raw vLLM/TRT-LLM when: You need a model that's not in the NIM catalog. You need custom configurations that NIM doesn't expose. You're running non-NVIDIA hardware. You want to experiment with different quantizations or engine parameters.

Chapter 7Training and Fine-Tuning: Making a Model Your Own

Everything up to this point has been about running models that someone else built. But what if the existing models aren't quite right for your use case? What if you need a model that writes in your company's style, knows your domain's terminology, or follows your specific output format? That's where fine-tuning comes in.

The Training Spectrum

There's a spectrum of how much you can customize a model, from least to most effort:

1. Prompting (zero effort). The simplest approach — you give the model instructions in the system prompt. "You are a legal assistant. Always cite case numbers. Use formal language." This works surprisingly well for many use cases and costs nothing. Always try this first.

2. Few-shot examples (minimal effort). Include examples of the input-output pairs you want in your prompt. "Here are three examples of how to format a customer response: [examples]. Now do the same for this customer." The model learns the pattern from your examples. No training required.

3. RAG — Retrieval-Augmented Generation (moderate effort). Give the model access to your documents. Instead of training the knowledge into the model, you feed relevant documents into the context at query time. This is what Open WebUI's document upload does. The model stays general but has access to specific knowledge. Best for when the information changes frequently.

4. Fine-tuning (significant effort). Actually modify the model's weights using your data. The model doesn't just have instructions or examples — it has been trained to behave the way you want. This produces the most consistent results but requires data, compute, and technical knowledge.

5. Pre-training from scratch (massive effort). Train a model from zero. This is what Meta does with Llama, what Alibaba does with Qwen. It requires millions of dollars in compute, terabytes of data, and months of work. Unless you're a well-funded research lab, this isn't for you.

When Fine-Tuning Makes Sense

Fine-tuning is powerful but expensive (in time and compute). Before committing to it, ask:

Have you tried prompting? A well-written system prompt with examples solves 70% of customization needs. If you haven't optimized your prompt, do that first. It's free and instant.

Have you tried RAG? If the issue is that the model doesn't know domain-specific information (your products, your policies, your documentation), RAG is usually better than fine-tuning. RAG updates instantly when your documents change; a fine-tuned model is frozen in time.

Is the behavior consistent? Fine-tuning shines when you need the model to consistently follow a specific pattern that's hard to describe in a prompt. Things like: always outputting valid JSON in a specific schema, following a complex multi-step protocol, or matching a very specific writing style across thousands of outputs.

Fine-Tuning in Practice

If you've decided fine-tuning is right for your use case, here's what the process looks like:

1. Prepare your data. You need examples of the input-output pairs you want the model to learn. Typically in JSONL format — each line is a conversation with messages. Quality matters more than quantity. 100 excellent examples beat 10,000 mediocre ones. Aim for diversity — cover the range of inputs the model will see.

2. Choose a base model. You're not training from scratch — you're taking an existing model (like Qwen 2.5 7B or Llama 3.1 8B) and adjusting it with your data. Pick a model that's already good at the general task you need. A coding model for code tasks. A multilingual model if you need multiple languages.

3. Choose a fine-tuning method. Full fine-tuning modifies all the model's weights — effective but needs a lot of GPU memory. LoRA (Low-Rank Adaptation) modifies a small subset of weights — much cheaper, almost as effective, and you can swap LoRA adapters in and out like plugins. For most people, LoRA is the right choice.

4. Train. Tools like Unsloth (fastest, easiest), Axolotl (flexible), or Hugging Face TRL (official) handle the training loop. On a single consumer GPU, fine-tuning with LoRA on 1,000 examples takes 1–4 hours depending on model size. Cloud options (Lambda Labs, RunPod, Vast.ai) rent GPUs by the hour if your local hardware isn't enough.

5. Evaluate. Test the fine-tuned model against your holdout examples. Compare it to the base model with good prompting. If fine-tuning isn't significantly better, your data might need improvement.

Tools for Fine-Tuning

Unsloth: The fastest and most beginner-friendly option. Optimized for consumer GPUs. Supports LoRA and QLoRA (quantized LoRA — fine-tunes a quantized model, needing even less memory). Can fine-tune a 7B model on a 24GB GPU in under an hour.

Hugging Face AutoTrain: A no-code/low-code option. Upload your dataset, pick a model, click train. Good for people who want results without learning the training pipeline.

Axolotl: A configuration-driven training framework. More complex than Unsloth but more flexible. Good for advanced users who need fine control over the training process.

Cloud fine-tuning: OpenAI, Anthropic, Google, and Mistral all offer fine-tuning as a service — upload your data and they train on their infrastructure. More expensive than doing it yourself, but zero setup required.

Chapter 8The Art of Prompting

Every interaction with an AI starts with a prompt — the text you give it. The quality of that prompt determines the quality of the response more than almost any other factor. A mediocre model with a great prompt often outperforms a great model with a mediocre prompt.

Prompting isn't magic. It's communication. And like all communication, it can be learned, practiced, and improved.

The Anatomy of a Good Prompt

Every effective prompt has four components, whether you state them explicitly or not:

1. Role — Who should the AI be? "You are a senior Python developer with 15 years of experience in web APIs." This isn't just flavor text. It changes the model's behavior. A "senior developer" writes differently from a "beginner-friendly tutor." A "legal expert" analyzes differently from a "general assistant." The role sets the frame for everything that follows.

2. Context — What does the AI need to know? Background information that's essential for a good response. "I'm building a FastAPI application with PostgreSQL. The current error occurs when processing requests with Unicode characters in the email field." The more relevant context you provide, the less the model has to guess — and guessing is where hallucinations come from.

3. Task — What should the AI do? Be specific. "Fix the bug" is vague. "Find why the email validation regex rejects addresses containing a plus sign, and propose a fix that passes RFC 5322" is actionable. The task should be clear enough that you could hand it to a skilled human and get the right result.

4. Format — How should the output look? "Respond with a JSON object containing 'diagnosis' and 'fix' keys." "Use bullet points, not paragraphs." "Give me a Bash script, not a Python script." If you don't specify format, the model picks one — and it might not be what you want.

System Prompts vs User Prompts

Most AI APIs distinguish between a system prompt (instructions that persist across the entire conversation) and user prompts (individual messages). This distinction matters.

The system prompt is where you put the role, persistent rules, and output format preferences. It's read once and influences every response. Think of it as the employee handbook — it sets expectations for the entire engagement.

The user prompt is where you put the specific task and context for each interaction. It changes every turn. Think of it as the work request — it says what needs to be done right now.

Good system prompt: "You are a DevOps engineer at a small IT consultancy. You write infrastructure-as-code using Terraform and Docker. You prefer pragmatic solutions over theoretical perfection. When you suggest changes, explain why. Always consider cost implications."

This prompt doesn't describe a task — it describes a way of thinking. Every response the model gives will be colored by these instructions.

Advanced Prompting Techniques

Chain-of-thought (CoT). Ask the model to think step by step before answering. "First, analyze the error. Then identify possible causes. Then evaluate each cause. Finally, recommend a fix." This forces the model to reason through the problem rather than jumping to a conclusion. It dramatically improves accuracy on complex tasks.

Few-shot examples. Show the model what you want by providing examples. "Here's how I want you to format customer responses: [example 1]. [example 2]. Now respond to this customer using the same format." The model learns the pattern from your examples — tone, structure, level of detail — without explicit rules.

Negative prompting. Tell the model what not to do. "Do not start with 'Great question!' Do not use filler phrases. Do not explain things I already know." Models have habits. Negative prompting breaks them. This is especially effective when the default behavior is annoyingly chatty or overly cautious.

Structured output. Ask for specific formats. "Respond with valid JSON matching this schema: {diagnosis: string, severity: 'low'|'medium'|'high', fix: string}." Many models can produce structured output reliably if you tell them the exact format. This is essential for automation — when another program needs to parse the output.

Constraint-based prompting. Set explicit limits. "In 3 sentences or fewer, explain..." or "Using only standard library functions, write a script that..." Constraints force the model to be concise and creative. Without them, models tend to be verbose — they'll write 500 words when 50 would do.

Chapter 9Prompt Caching and Cost Control

When you use cloud AI APIs, you pay per token — both for the tokens you send (input) and the tokens you receive (output). On long conversations with large system prompts, this adds up fast. Prompt caching is a technique that can cut your costs dramatically.

How Prompt Caching Works

Every time you send a request to a cloud model, the API processes your entire prompt — system instructions, conversation history, everything. If your system prompt is 2,000 tokens and you send 50 messages, that's 100,000 tokens just in system prompt repetition. You're paying to process the same instructions over and over.

Prompt caching solves this by letting the API remember the static parts of your prompt. The first request processes everything normally. Subsequent requests that start with the same prefix reuse the cached computation. The result: cached tokens cost 90% less than fresh tokens on most providers.

Anthropic's Claude, for example, charges $0.30/MTok for cached input tokens vs $3.00/MTok for regular input. That's a 10× reduction. For a system with four agents, each processing dozens of messages per day with 2,000-token system prompts, the savings are substantial — easily hundreds of dollars per month.

What Gets Cached

System prompts. The biggest win. Your system instructions don't change between turns, so they're cached after the first request. This alone can cut costs 30–50% on long conversations.

Conversation history. The earlier messages in a conversation are the same on every turn — only the latest message is new. Caching handles this automatically. A 100-turn conversation doesn't process all 100 turns fresh each time — only the new content.

Tool definitions. If your AI has MCP tools or function definitions, those are part of the prompt too. They get cached along with the system prompt.

How to Use It

The good news: on most modern APIs, caching happens automatically. You don't need to do anything special — the provider detects repeated prefixes and caches them. Anthropic, OpenAI, and Google all support some form of prompt caching.

The key optimization you can make: put static content first. Since caching works on prefixes (the beginning of the prompt), structure your requests so the unchanging parts come before the changing parts:

If you put the dynamic content first and static content later, nothing gets cached because the prefix changes every time.

Beyond Caching: Other Cost Controls

Max tokens. Always set a max_tokens limit. Without it, a model can generate thousands of tokens on a simple question. Setting max_tokens: 500 for routine tasks prevents runaway costs.

Model selection. Use the smallest model that handles the task. Claude Haiku at $0.25/MTok input vs Claude Opus at $15/MTok is a 60× cost difference. For simple extraction, summarization, or classification, the smaller model is fine. Save the big model for complex reasoning.

Local fallback. Route routine tasks to local models (free) and use cloud APIs only for tasks that need them. We covered this architecture in Chapter 13 — it's not just about speed, it's about cost.

Token monitoring. Track your usage. Every API provides usage data in the response headers or body. Log it. Dashboard it. Set alerts. The first step to controlling costs is seeing them.

Chapter 10When to Enable Thinking Mode

Some models have a thinking mode (also called "extended thinking" or "reasoning mode") — a feature where the model explicitly reasons through the problem before producing its answer. Instead of jumping straight to a response, it generates an internal chain of thought, works through the logic, considers alternatives, and then gives you its conclusion.

This isn't just a gimmick. On complex problems, thinking mode can be the difference between a correct answer and a plausible-sounding wrong one. But it comes with tradeoffs that you need to understand.

How Thinking Mode Works

When you enable thinking, the model generates two streams: a reasoning trace (the "thinking") and the final response. The reasoning trace is where the model works through the problem — analyzing constraints, considering edge cases, evaluating options, correcting mistakes. The final response is the polished answer.

In some implementations (like Claude's extended thinking), you can see the reasoning trace. In others (like Nemotron-Nano's built-in CoT), the reasoning happens in a hidden field that you don't see but that influences the output quality. Either way, the model is doing more work, and the results show it.

When Thinking Mode Helps

Complex reasoning. Multi-step math, logic puzzles, code architecture decisions. Tasks where the answer requires building on intermediate conclusions. Without thinking mode, the model might skip steps and arrive at a wrong answer that sounds right.

Code debugging. "Here's a 200-line function with a subtle bug." Thinking mode lets the model trace through the logic systematically rather than pattern-matching to a likely fix. The difference in accuracy is significant.

Planning and strategy. "Design a migration plan for moving 9 services to Cloudflare." The model needs to consider dependencies, ordering, risks, and fallbacks. Thinking mode ensures it actually works through these considerations rather than generating a generic checklist.

Ambiguous prompts. When the task isn't clear, thinking mode helps the model identify the ambiguity and resolve it rather than picking one interpretation and running with it.

When Thinking Mode Hurts

Simple tasks. "What's the capital of France?" doesn't need chain-of-thought reasoning. Thinking mode on trivial questions just wastes tokens and time.

Speed-critical applications. Thinking mode is slower — sometimes 2–5× slower. If your use case is a chatbot that needs to respond in under a second, thinking mode adds unacceptable latency.

Token-limited contexts. Here's the trap we discovered in our benchmarks: reasoning tokens count against your token budget. If you set max_tokens: 500 and the model spends 400 tokens thinking, you only get 100 tokens of actual response. The output gets truncated mid-sentence. We learned this the hard way with Nemotron-Nano — complex prompts with low token limits produced empty responses because the reasoning consumed the entire budget.

High-volume inference. If you're processing 10,000 documents, thinking mode on each one multiplies your cost and time. Use it on a sample to validate your approach, then turn it off for the batch.

The Practical Rule

Chapter 11Benchmarks: What We Actually Measured

Theory is one thing. We ran real benchmarks on our hardware — two NVIDIA DGX Spark machines with GB10 Blackwell chips — comparing models head-to-head across different tasks and perspectives. Here's what we found, and what it taught us about choosing models for real work.

The Setup

We tested two models running on TensorRT-LLM with FP4 quantization:

— Nemotron-Nano (8B parameters, reasoning-enabled) on the "spark" node
— Llama-4-Scout-17B (17B parameters, standard) on the "dark" node

Same hardware class, same inference engine, same quantization. The variable was the model itself.

We didn't just run one test — we ran four different benchmarks, each designed by a different agent with a different perspective. The coordinator tested communication and planning. The strategist tested business reasoning. The technocrat tested infrastructure code generation. The generalist tested cross-domain reasoning depth.

The Results

What We Learned

The smaller reasoning model was consistently faster. Nemotron-Nano (8B) was 2–4× faster than Llama-4-Scout (17B) on every test. This wasn't expected — smaller models are generally faster, but not by this much. The reason: Nemotron's architecture is optimized for efficient inference, and TRT-LLM exploits this well on Blackwell hardware.

Built-in reasoning changes the game. Nemotron-Nano has chain-of-thought reasoning baked in — it thinks before it answers, in a hidden reasoning field. This produced noticeably higher-quality structured output. When asked about ambiguity, Nemotron identified and addressed the ambiguity. Llama-4-Scout just picked an interpretation and went with it. For any task requiring judgment, the reasoning model was clearly superior.

But reasoning has a cost. Nemotron's reasoning tokens eat into the max_tokens budget. On our first infrastructure benchmark with 700 token limit, Nemotron produced an empty response — the entire budget was consumed by internal reasoning. We had to increase to 1,200+ tokens to get usable output. This is a critical gotcha: reasoning models need at least 2× the token budget you'd give a non-reasoning model.

Different models for different tasks. The four perspectives revealed something a single test wouldn't: Nemotron excels at deep analysis of one dimension (expert-level reasoning on a single topic), while Llama-4-Scout is better at comprehensive coverage (hitting all sections of a multi-part prompt). For research and planning, use the reasoning model. For checklists and thorough documentation, the standard model is more reliable.

The council consensus:

How to Run Your Own Benchmarks

Don't just trust published benchmarks. They measure generic tasks on standard datasets. Your use case is specific. Here's how to build a useful benchmark for your own work:

1. Use real tasks. Take 5–10 actual prompts from your daily work. Not toy examples — real questions you've asked AI in the last week. This is the most honest test of model quality for your use case.

2. Test from multiple perspectives. This is where multi-agent thinking helps. A prompt that looks great from a technical perspective might produce business nonsense. Have someone from a different background evaluate the output.

3. Measure what matters. Speed (time to first token, total time). Quality (does the answer actually help?). Consistency (same quality on the 50th run as the 1st?). Token efficiency (how much of the output is filler?). Pick 2–3 metrics that matter for your use case.

4. Test edge cases. Give the model an ambiguous prompt. Give it a long prompt. Give it a prompt in a domain it probably hasn't seen much. These edge cases reveal more about model quality than standard benchmarks.

Chapter 12The Web Interface That Changes Everything

Talking to a model in the terminal is fine for testing. But for real work, you want Open WebUI.

Open WebUI is a self-hosted web interface that connects to your Ollama models and transforms them from a command-line curiosity into a full AI workstation. It's free, open source, and installs with a single Docker command.

Open http://localhost:3000 in your browser. That's your AI workstation.

Why This Is a Game-Changer

Document upload and RAG. Drag a PDF, Word document, or text file into the chat. The AI reads it and answers questions about it. This alone is worth the setup — you can analyze contracts, summarize research papers, or query technical documentation without sending anything to the cloud. RAG (Retrieval-Augmented Generation) means the AI doesn't just read the document once; it indexes it and retrieves relevant sections as you ask questions.

Model switching mid-conversation. Start a conversation with a fast, small model for brainstorming. Switch to a larger model when you need deeper analysis. Switch to a vision model when you need to analyze an image. All in the same conversation thread, without losing context.

Conversation history and search. Every conversation is saved. You can search across all your past chats. Three weeks from now, when you think "I solved this problem before," you can find exactly what you did.

Voice input and output. Talk to your AI instead of typing. This sounds like a gimmick until you're debugging at 2 AM and your hands are tired. It also makes AI accessible to people who aren't comfortable typing long prompts.

Multiple users. Open WebUI supports user accounts. You can give team members their own login, their own conversation history, their own model preferences. One installation serves a whole team.

Chapter 13Give AI Hands with MCP

Up to this point, your AI can think and talk. It can analyze documents and write code. But it can't do anything. It can tell you what database query to run, but it can't run it. It can write a script, but it can't execute it. It can describe what's wrong with your server, but it can't SSH in and fix it.

MCP — the Model Context Protocol — changes that. It's a standard (created by Anthropic and now widely adopted) that lets AI models call external tools. Think of it as giving the AI hands.

How MCP Works

The concept is simple. An MCP server is a small program that exposes a set of tools — things like "read a file," "query a database," "search the web," "create a git commit." The AI can see what tools are available and decide when to use them. When it calls a tool, the MCP server executes the action and returns the result.

For example: you ask your AI "what's the disk usage on the server?" Without MCP, it tells you to run df -h. With MCP, it runs df -h, reads the output, and tells you "you have 23% free on /dev/sda1, but /var/log is 94% full — you should rotate the logs."

The difference is transformative. The AI goes from advisor to operator.

Getting Started with MCP

Pre-built MCP servers exist for most common tools. Filesystem operations, Git, PostgreSQL, MySQL, web search, Slack, and dozens more. The MCP ecosystem is growing rapidly — check mcp.so or search GitHub for "mcp server" plus whatever tool you use.

Docker Desktop's MCP Toolkit bundles many integrations and makes setup easy. If you use Docker (and you should), this is the fastest way to give your AI access to a broad set of tools.

Building your own MCP server is straightforward. If you have a REST API, a command-line tool, or any programmable interface, you can wrap it in an MCP server in a few hours. The protocol is simple — it's essentially JSON over HTTP/stdio.

Chapter 14The IDE as Mission Control

At some point you'll want to build something real — an application, a service, an automation. This is where AI-integrated development environments become essential.

Claude Code in VS Code (or similar tools like GitHub Copilot, Cursor, or Windsurf) turns your editor into a collaborative workspace. The AI isn't a separate window you paste code into. It's in your editor, reading your files, understanding your project structure, and writing code that fits your patterns.

What Changes When AI Lives in Your Editor

Context is automatic. When you chat with AI in a browser, you spend half your time explaining context: "I have a Python file that does X, and a config file that says Y, and the error is Z." In an IDE, the AI already sees all of that. It reads your project. It knows what frameworks you're using, what your functions do, what your tests expect. You skip the explanation and go straight to the problem.

Changes happen in-place. Instead of the AI showing you code in a chat window that you then copy-paste into your file, it edits the file directly. You see the diff. You approve or reject. The feedback loop is seconds, not minutes.

Multi-file awareness. Real projects span dozens of files. A change in the API endpoint affects the frontend call, the test, and the documentation. AI in the IDE can trace these connections and update everything consistently. In a chat window, you'd have to manually tell it about each file.

The conversation persists. Your IDE session remembers what you've been working on. "Remember that bug we fixed yesterday in the auth module? I think it's back, but in the payment flow." The AI has the context. It can check.

The CLAUDE.md Pattern

One of the most powerful patterns we discovered: create a file called CLAUDE.md (or AGENTS.md) in your project root. Write down your project conventions, architecture decisions, common pitfalls, and coding standards. The AI reads this file at the start of every session.

This is your project's institutional memory. Instead of repeating "we use snake_case for database columns" or "never import directly from the internal module" every time, you write it once. The AI follows it every session. New team members — human or AI — read it and get up to speed immediately.

Chapter 15Why One AI Isn't Enough

This is the idea that changed everything for us. And it's the one most people haven't tried yet.

A single AI assistant, no matter how capable, has one perspective. It approaches every problem the same way. Ask it a technical question, it gives a technical answer. Ask it a business question, it gives a business answer. But it rarely stops to ask: "Is this the right question? What are we missing? Who else should weigh in?"

Humans solved this problem thousands of years ago. We put different people in a room — the engineer, the accountant, the designer, the customer — and let them argue. The engineer says "this is technically elegant." The accountant says "this costs too much." The designer says "nobody can use this." The customer says "I just want it to work." The result is better than any single perspective could produce.

You can do the same thing with AI. And you should.

The Single-Perspective Trap

Consider a real scenario: you're planning to deploy a new service. A single AI assistant will help you write the Docker Compose file, set up the networking, configure the environment variables. It's technically competent. But it won't ask:

— "What's the cost of running this 24/7?" (business perspective)
— "What happens when it crashes at 3 AM and nobody's awake?" (operations perspective)
— "Does the client actually need this feature, or are we gold-plating?" (strategic perspective)
— "Is this documented well enough that someone else can maintain it?" (sustainability perspective)

A single agent answers the question you asked. Multiple agents question the question itself.

How Multi-Perspective AI Works in Practice

The setup is simpler than it sounds. You create multiple AI agents, each with a different system prompt that defines their personality and priorities. They share access to the same information — your project files, your documentation, your chat channels — but they process it through different lenses.

When you post a question or a plan to a shared channel, each agent responds from their perspective. You don't get one answer. You get a discussion. And in that discussion, blind spots get caught. Assumptions get questioned. Risks get identified before they become problems.

That multi-agent discussion just caught three problems that a single agent would have missed — not because the single agent is dumb, but because it wasn't thinking about cost, monitoring, or documentation. It was thinking about deployment, because that's what you asked about.

The Disagreement Is the Feature

Most people's instinct is to make their AI agents agree. They want harmony. Efficiency. No friction.

That's backwards. If all your agents always agree, they're not adding value. You're just running the same perspective four times. The value comes from disagreement — from the moment when the business agent says "this costs too much" and the technical agent says "but this is the right architecture." Now you have a real decision to make, with real tradeoffs laid out clearly.

In human teams, this is called "constructive conflict." It's the reason boards of directors have people from different backgrounds, the reason design reviews include non-engineers, the reason good managers hire people who disagree with them. The same principle applies to AI teams.

Chapter 16Designing Agent Personalities

Setting up multiple agents isn't just about copying the same AI four times. Each agent needs a distinct identity — a set of priorities, a communication style, and a domain of concern that makes it genuinely different from the others.

The Four Perspectives That Cover Most Projects

These aren't the only possible roles. You might want a security-focused agent, a UX-focused agent, or a compliance-focused agent. The point is that each role represents a way of thinking, not just a skill set.

Personality Is More Than a System Prompt

A good agent personality has three layers:

1. Priorities. What does this agent care about most? The strategist cares about money and customers. The technocrat cares about reliability and security. These priorities determine what the agent notices and what it ignores. An agent without clear priorities is just a generic assistant wearing a hat.

2. Communication style. How does this agent talk? The coordinator is concise and action-oriented ("Here's the plan, here's who does what, go"). The strategist frames everything in terms of impact ("This saves the client 4 hours/week, which justifies the $40/month"). The technocrat is precise and cautious ("This works, but the connection pool is set to 20 and we have 50 concurrent users — that's a bottleneck at scale"). Different styles surface different information.

3. Memory and continuity. Each agent should have its own memory — its own history of what it's seen, what it's learned, and what mistakes it's made. An agent that remembers "last time we skipped the health check and the service went down for 6 hours" is more valuable than one that approaches every deployment fresh. Memory creates judgment. Judgment creates genuine perspective.

How to Start

Don't start with four agents. Start with one. Give it a clear role and personality. Use it for a week. Pay attention to the moments where you wish you had a second opinion — "I wish someone would check the cost" or "I wish someone would think about the edge cases." Those wishes tell you what your second agent should focus on.

Add agents one at a time. Each new agent should fill a gap you've actually experienced, not a gap you theorize about. Two well-designed agents are more valuable than four generic ones.

Chapter 17Architecture: Cloud Brain, Local Hands

Here's the architecture that works in practice: use cloud AI for thinking and local hardware for doing.

This isn't just a cost optimization. It's a fundamental design principle that determines what's fast, what's private, and what's resilient.

Why This Split Matters

Cost control. Cloud AI is priced per token. Every time an agent thinks, you pay. But most of an agent's work isn't thinking — it's executing: running commands, processing data, generating text from local models. If you route execution to local hardware, you only pay for the orchestration layer. In our setup, the cloud AI decides what to do, and local hardware does it.

Data privacy. When your agents SSH into a server and query a database, the data flows from your database to your local agent to your local GPU. It never leaves your network. The cloud AI only sees the orchestration: "query the database for customer counts" goes up; "query returned 847 rows" comes back. The actual data stays local.

Speed. An agent running a command on a local server gets millisecond latency. A cloud-only agent adds a network round-trip for every operation. When an agent is doing a complex task — reading logs, running queries, updating configs — those round-trips multiply. Local execution is simply faster for operational work.

Resilience. If the cloud API goes down (and it will — every API has outages), your local infrastructure still works. Local models can still run inference. Agents can still execute tasks using their local capabilities. The cloud is the brain; the local hardware is the body. The body can still function, at reduced capacity, when the brain is temporarily offline.

The Hard Rule

Always explicitly specify which model runs where. This is the single most important operational lesson we learned. If your agents default to the cloud API when you meant them to use local GPUs, you'll burn through credits fast. Set the default model for every agent. Enforce it in configuration. Check your logs to verify it's working.

We lost $78 in one hour because spawned agents silently fell back to cloud inference instead of using the local GPUs. That's not a lot of money in absolute terms, but it represents an architecture failure — the boundary between "cloud brain" and "local hands" wasn't enforced. After that, we built explicit routing: cloud for orchestration, local for everything else. No exceptions. No defaults that could surprise us.

Chapter 18Mistakes That Cost Money (So Yours Don't)

Every lesson here was learned the hard way. We're sharing them so your hard way is cheaper.

1. Tokens Add Up Faster Than You Think A single long conversation with a cloud model can cost $1–5. That doesn't sound like much until you have four agents, each having multiple conversations per day, running for a month. Track your spending from day one. Set budget alerts. Review usage weekly. The first time you see a $200 invoice you weren't expecting, you'll wish you'd started monitoring earlier.
2. Session Bloat Kills Performance Every message in a conversation adds to the context. AI models have a context window — a maximum amount of text they can "see" at once. When your conversation history fills that window, the model slows down, misses relevant information, or starts hallucinating. Reset long conversations periodically. Start fresh sessions for new topics. Archive important context in files the AI can read, rather than keeping it in chat history.
3. Silent Failures Are Worse Than Loud Ones An agent that crashes tells you something is wrong. An agent that quietly produces wrong answers, or goes silent when it should respond, can go unnoticed for hours. Build monitoring. Check logs. If an agent hasn't responded in a channel for an unusual amount of time, that's a signal. Treat silence as suspicious.
4. API Keys Expire Quietly Authentication tokens have expiry dates. Your entire system works perfectly on Monday. On Friday, a token expires, an agent can't reach its API, and everything downstream breaks. The error message is usually something unhelpful like "unauthorized" or "connection refused." Keep a calendar of token expiry dates. Set up monitoring that checks auth health. Better yet, implement automatic token refresh where possible.
5. Don't Edit Configs by Hand Tools that manage configuration files often cache them in memory. If you edit the file directly, the tool doesn't notice — it keeps using the cached version. You think you've made a change; the system disagrees. Always use the tool's official CLI or API to modify configuration. We spent hours debugging a "broken" config that was actually fine — the gateway just hadn't reloaded it.
6. Test One Thing at a Time When multiple things change simultaneously, you can't tell which change caused a problem. Deploy one change. Test it. Confirm it works. Then deploy the next change. This is obvious advice that everyone ignores until they're debugging a system where three things changed and any of them could be the cause.

Chapter 19Your First 90 Days

Here's a realistic roadmap for going from "interested in AI" to "running a multi-agent production system." It's the path we wish someone had given us.

1. Week 1–2: Learn to Talk to AI Sign up for a cloud AI (Claude, ChatGPT, Gemini — any of them). Use it daily. Not for novelty — for actual work. Summarize documents, draft emails, debug code, brainstorm ideas. The goal is to build intuition for what AI is good at (synthesis, generation, pattern matching) and what it's bad at (precise math, real-time information, consistent long-term memory). This intuition will guide every decision you make later.
2. Week 3–4: Put AI in Your Editor Install Claude Code, Copilot, Cursor, or Windsurf in your editor. Start a small project — a script, a simple web app, an automation tool. Experience the difference between asking an AI about code in a chat window and working with an AI that lives in your codebase. Create a CLAUDE.md file with your project conventions. Notice how much faster the second week is compared to the first, because the AI has learned your patterns.
3. Week 5–6: Go Local Install Ollama. Download Qwen 2.5 (7B). Run your first local model. Install Open WebUI for a proper interface. Upload a document and query it. Compare the experience to cloud AI — notice what's better (privacy, cost, freedom) and what's worse (speed, capability). Start using local models for routine tasks and cloud models for complex ones.
4. Week 7–8: Give AI Tools Set up your first MCP server — start with filesystem access. Let the AI read and navigate your project files. Then add a database connection. Watch the AI go from "here's what you should query" to "I queried it and here are the results." Add tools incrementally. Read-only first, write access later.
5. Month 3: Add Perspectives Create a second AI agent with a different focus — if your first agent is technical, make the second one business-oriented. Give them access to a shared channel. Post a plan and watch them respond from different angles. Notice the moments where the second perspective catches something the first one missed. That's the value. Add more agents only when you feel a specific gap.
6. Month 3+: Build Your Architecture Set up the cloud brain / local hands split. Route orchestration through cloud AI, execution through local hardware. Configure model defaults explicitly. Build monitoring. Document everything. You now have a system, not just a tool. Maintain it like infrastructure — because that's what it is.