Multimodal and Code Agents

Why this week is positioned where it is. Weeks 5-7 built RAG and agent systems. Week 8 gave eval discipline. Week 9 covered inference cost. Multimodal and code agents inherit all of those concerns and amplify them. Multimodal makes the cost equation 5-10× harsher. Code agents are agents (Week 7) running against real-world systems with real failure modes. Both are at the production frontier in 2026 — the techniques work, but the engineering is unforgiving.

1. Multimodal foundations

The 2026 multimodal landscape has converged on a few architectural patterns:

Late-fusion (most common). A vision encoder (CLIP, SigLIP, or a custom ViT) converts the image to a sequence of "visual tokens" that get prepended to the text input. The LLM treats them as part of the input prompt. Used in Claude Vision, early GPT-4V, most open-source VLMs (LLaVA, InternVL, Qwen-VL).

Unified / native multimodal. The model is trained from scratch on interleaved text and other modalities. The architecture treats all modalities as token streams. GPT-5 (and GPT-4o before it), Gemini, and Anthropic's newer Claude models use variants of this. Higher quality on multimodal reasoning; harder to develop.

Side-channel adapters. Add modality-specific projection layers to a frozen LLM. Faster to develop, lower quality ceiling. Used in some research models and rare in production.

Vision token costs. A typical image becomes 1,000–2,000 tokens depending on resolution and model. The provider tokenizers vary:

• Claude (2026): ~85 tokens for a thumbnail (≤256px), ~768 tokens for a medium (≤1024px), ~1568 tokens for a high-detail (up to 1568×1568).
• GPT-5: ~85 tokens for "low detail," ~765 + 170 per 512px tile for "high detail."
• Gemini 2.5: ~258 tokens for ≤384px, scaling up to ~1024 tokens for higher resolution.

The rule of thumb: one image is roughly 1,000 tokens of input cost. This number drives the economics.

Resolution tradeoffs. Lower resolution = cheaper, faster, but loses fine detail (small text, distant features, dense charts). Higher resolution = better OCR-like extraction but expensive. The right resolution depends on your task: receipts often need high; "is this a cat or a dog?" works at low.

Audio token costs. ~600 tokens per minute for typical conversational audio in models that accept audio directly. Whisper produces text from audio at ~$0.006/minute and is still the cost leader for transcription-only workloads.

Video token costs. Sampling at 1 fps and treating each frame as an image: ~60,000 tokens per minute. At Sonnet pricing, that's $0.18 per minute of video input. Past 10 minutes of video, your costs become serious.

2. The economics of multimodal

The cost-economics step from text-only to multimodal is brutal and underestimated.

A reference comparison at Sonnet 5 prices ($3/M input, $15/M output):

• Text-only chat: 500-token prompt + 200-token response = $0.0045/call.
• One screenshot in: 500-token prompt + 1500-token image + 200-token response = $0.0090/call. 2× cost.
• 5 screenshots in: 500-token prompt + 7500-token images + 200-token response = $0.0270/call. 6× cost.
• 10-second video clip: 500-token prompt + 10,000-token frames + 300-token response = $0.0345/call. 7.7× cost.
• 1-minute video clip: 500-token prompt + 60,000-token frames + 500-token response = $0.1875/call. 42× cost.

The per-request multipliers compound at scale. A team that adds vision to their existing chat product without changing the model just 5-10×ed their cost.

Why this happens. Vision tokens are cheap to generate (the encoder is small) but expensive to process — they consume the same per-token attention compute as text tokens, and there are more of them. The LLM doesn't know they're "images" — they're just a longer prompt.

The optimization toolkit for multimodal is narrower than text:

• Resolution downscaling. First and largest lever. Most production tasks work at medium resolution.
• Image deduplication. If the same screenshot appears in 100 conversations (e.g., a recurring UI element), cache it. Anthropic's prompt caching covers this.
• Frame sampling for video. Instead of 1 fps, try 0.25 fps for slow-motion content. 4× cost reduction.
• Pre-extraction with OCR. For document-heavy workloads, OCR once with a cheap dedicated tool, then send only text to the LLM.
• Specialized models for narrow tasks. A receipt parser fine-tuned on receipts is 10-100× cheaper than a frontier VLM.

Multimodal output is rarely the bottleneck. Most multimodal workloads have small text outputs. Output cost is a small fraction.

3. Image understanding in production

The four most common production multimodal workloads:

Document understanding. Receipts, invoices, contracts, legal documents. The dominant production workload in 2026. Two architectures compete:

• Pure VLM: send the page directly to Claude/GPT-5/Gemini. Best quality on novel documents, handwriting, complex layouts. Most expensive.
• OCR-then-LLM: dedicated OCR (AWS Textract, Google Document AI, Mistral OCR) extracts text + structure, then a text-only LLM does understanding. 10-20× cheaper. Lower quality on the long tail of edge cases.

The 2026 sweet spot for high-volume document processing: tier-based routing. OCR for the 90% of standard pages; VLM for the 10% that confuses OCR (handwriting, complex layouts, charts within documents).

Screenshot / UI understanding. "Help me fill out this form," "What's wrong with this UI?", browser agents that act on visual state. VLM-only territory — no OCR pipeline gives you "understanding of layout and intent." Cost matters less because requests are interactive (lower volume).

Charts and graphs. Extracting data from charts. Genuinely hard. VLMs are the best path; quality varies dramatically by chart complexity. Pre-extracting with chart-specific models (DePlot, ChartQA-style) helps when applicable.

Photo / scene understanding. Product photos, real-world images, security footage. VLMs handle most of this well. Specialized models still win for very narrow tasks (face recognition, vehicle classification).

The OCR-vs-VLM decision tree:

1. Is the input mostly text in known formats (invoices, forms with consistent layouts)? → OCR.
2. Does the task require understanding layout, charts, or non-text visuals? → VLM.
3. Is volume > 100k pages/month? → OCR + LLM tier-routing or fine-tuned specialized model.
4. Is volume < 10k pages/month and quality is critical? → VLM directly.
5. Mixed → tier-based routing with VLM as fallback.

4. Audio and speech

The 2026 audio stack:

Whisper (still the standard). OpenAI's open-source model. Transcribes audio to text. Multilingual. ~$0.006/minute on hosted APIs; free if you self-host. Very mature. Used as the speech-to-text layer for almost everything.

Voicebox / TTS systems. Voice generation. Eleven Labs, OpenAI TTS, Resemble, Coqui. Voice cloning concerns are real and getting tighter regulation in 2026.

Audio-in models. GPT-5 audio, Gemini Live, Anthropic Claude with audio. Accept audio directly without transcription. Better than transcribe-then-LLM for prosody, emotion, multiple speakers. ~$0.024/minute for input audio at typical pricing.

Real-time voice. OpenAI Realtime API, Gemini Live. Sub-second latency end-to-end. The frontier of conversational AI. Cost: 5-10× higher than batch transcription.

The architecture decision:

• Transcribe-then-LLM. Whisper → text → LLM. Cheaper, slower (sequential), loses prosody info. Best for batch processing, transcripts, search.
• Audio-direct. Audio → multimodal LLM. More expensive, faster (parallel), preserves all signal. Best for real-time conversation, voice agents, multi-speaker analysis.

Pick by use case: customer support recordings → transcribe; voice assistants → audio-direct.

Production audio gotchas:

• Diarization (who said what) is hard. Most APIs handle 2-3 speakers reasonably; struggle past 5.
• Background noise kills accuracy. Production deployments need preprocessing.
• Long audio (1hr+ files) often requires chunking with overlap to avoid context loss.
• Streaming vs batch is a different API surface; build for one or the other, not both.

5. Video — the cost frontier

Video is the most expensive multimodal modality, by a lot.

The naive approach: frame sampling. Extract frames at 1 fps, send each to a VLM. Costs scale linearly with video length and inversely with sampling rate.

• 1 fps × 1 minute = 60 frames × ~1000 tokens = 60k input tokens
• At Sonnet pricing: $0.18 per minute of video input

Realistic costs for video workloads:

• 30-second TikTok: $0.10-0.20 per video.
• 5-minute YouTube tutorial: $0.90-1.50 per video.
• 1-hour podcast (with video): $11-22 per podcast.
• 8-hour security footage day: $90-180 per day per camera.

For most video applications, this is too expensive at scale.

The production toolkit for video:

• Lower frame rate. Most non-action content works at 0.25 fps (4× savings). Static content at 0.05 fps.
• Audio-only first pass. Transcribe audio with Whisper ($0.006/min). Use the transcript to identify "moments of interest." Sample frames only around those moments.
• Scene detection. Use cheap CV models (PySceneDetect) to find scene boundaries. Sample 1-2 frames per scene instead of 1 per second.
• Specialized models. For narrow tasks (action recognition, OCR in video), specialized lightweight models are 100× cheaper than VLMs.
• Native video models. Gemini 2.5 and some research models accept video directly with more efficient temporal compression. Still expensive but better than naive frame sampling.

The 2026 reality: video is mostly a research and high-end-product workload. Mass-market video understanding is constrained by cost. Workflows that ship in production usually have audio-first architectures with sparse frame sampling.

6. Code agents — the production frontier

Code agents are agents (Week 7) operating against real codebases with real tooling. The 2026 leading systems — Claude Code, Cursor, Windsurf, Devin, GitHub Copilot's agent mode — share a common architecture pattern.

The canonical loop:

1. Read the user's task description.
2. Explore the codebase: read relevant files, understand the structure.
3. Plan the change.
4. Make edits.
5. Run tests.
6. Iterate on failures.
7. Submit / commit.

The tools that matter:

• read_file(path) — load file contents into context.
• edit_file(path, old, new) or write_file(path, content) — modify code.
• bash(cmd) — run arbitrary commands in a sandbox. Critical for tests, builds, language servers.
• grep / search — find symbols, files, patterns.
• ls(path) — directory listing.
• run_tests — typed shortcut for the test command.

Sandbox is non-negotiable. Code agents that run arbitrary bash commands need an isolated environment. Production code agents use Docker containers or dedicated VMs, never the user's actual machine for arbitrary commands. Security and reproducibility both demand this.

Long context matters more here than anywhere else. A real codebase is millions of tokens. Agents need:

• File-level retrieval. Vector search or grep to find relevant files first.
• Hierarchical summarization. Read directory structures, then files within selected directories.
• Context window management. Drop earlier turns when context gets long. Compress. Re-summarize.

The "pause and ask" pattern. Production code agents in 2026 ask before destructive actions: deleting files, running migrations, force-pushing. The trust model is "agent suggests, human approves, agent executes." Fully autonomous agents on production codebases remain rare.

What separates good code agents from bad:

1. Tool design. Clean, typed tools with helpful error messages. The Week 7 lessons apply double here.
2. Prompting. System prompts that encode senior-engineer practices: read before editing, run tests after every change, prefer minimal diffs.
3. Context management. Aggressive trimming. Summary of older turns. File contents loaded lazily.
4. Grounding. Always read the actual code before reasoning about it. Don't reason from memory of similar code.
5. Verification loop. After every meaningful change, run tests. If fail, iterate. Don't claim success without verification.

7. SWE-bench and reading code benchmarks honestly

SWE-bench is the dominant code-agent benchmark. The construction:

• ~12,000 tasks from real GitHub issues across popular Python repos (Django, sympy, scikit-learn, etc.)
• Each task: starting state, problem statement, hidden test cases that verify the fix.
• Score: % of tasks where the agent's patch passes the hidden tests.

SWE-bench Verified (2024) is a curated subset of ~500 tasks where humans confirmed the test cases are correct and unambiguous. The honest benchmark.

2026 leaderboard reality:

• Frontier code agents (Claude Code with Opus 5, Cursor with GPT-5.6 Sol) score 75-82% on SWE-bench Verified.
• Mid-tier (Sonnet 5, GPT-5.6 Terra): 60-70%.
• Budget agents: 40-55%.
• Human baseline: ~95% with unlimited time, ~75% in time-constrained settings.

How to read these numbers honestly:

• SWE-bench is biased toward Python and Django-style codebases. Performance on TypeScript / Go / Rust / C++ may differ significantly.
• Tasks are real but curated. The benchmark filters for tasks with clear test cases. Real engineering work has fuzzier success criteria.
• The metric is binary per task (patch passes or fails). Most production code-review and refactor tasks aren't binary.
• Tooling matters more than model. A weaker model with better tools (file search, language server) often beats a stronger model with naive tools.
• Cost varies wildly. A 75% agent that costs $5/task vs a 70% agent that costs $0.50/task — the cheaper one ships in production.

Don't pick a code agent based on SWE-bench score alone. Pick based on:

1. Performance on your code (build a small private eval set — Week 8).
2. Cost per task at your traffic level.
3. Tool ecosystem fit (does it work with your CI? IDE? language?).
4. Failure modes (graceful retries vs catastrophic edits).
5. Trust model (asks before destructive actions vs YOLO mode).

8. Production patterns

The 2026 production patterns that have stabilized:

For multimodal:

• Pure-VLM is for prototyping. Production at scale is OCR + LLM with VLM as fallback for hard cases.
• Cache aggressively. Recurring images (logos, UI elements, repeated screenshots) cache well. The cost is in the unique images.
• Resolution-by-task. Don't default to high-res. Calibrate the lowest resolution that holds quality on your eval set.
• Specialized models for narrow tasks. Receipt parsers, chart extractors, screenshot classifiers — fine-tunes on narrow tasks beat frontier VLMs on cost by 10-100×.
• Audio-first for video where possible. Transcribe with Whisper, find moments of interest in transcript, sparse-sample frames.

For code agents:

• Sandbox everything. Never run agent-generated code on the user's machine without a sandbox boundary.
• Git is the trust mechanism. Agents work in branches; humans review PRs. Production never bypasses code review.
• Test-driven loops. Every meaningful change must run tests. No exceptions.
• Aggressive context management. Summarize older turns. Drop file contents that aren't actively relevant. Re-load on demand.
• Cost monitoring per task. A code agent that costs $50/task because it loops is a bug, not a feature. Set per-task budgets.
• Trust progression. Start with read-only operations (suggest, don't apply). Move to PR-creation (suggest patches as PRs). Only progress to direct commits with strong verification.

The combined pattern: multimodal code agents. UI-aware code agents that can take screenshots of broken pages, read error messages from images, generate code based on Figma designs. Frontier in 2026; production in 2027 probably. Browser-using agents are the current rough state of the art (Claude with Computer Use, OpenAI Operator).

The cost discipline reminder. Both multimodal and code agents amplify the cost equation from Week 9. A code agent that takes 20 turns at 50k tokens each at Opus pricing costs $25/task. The same task with Sonnet and tighter context: $5. The same with Haiku for routing and Sonnet for hard parts: $1.50. The optimization order from Week 9 matters more here, not less.

Build this week

Pick at least two:

1. Build a document understanding pipeline. Take a stack of 100 invoices (or any structured docs). Build OCR-then-LLM and pure-VLM versions. Compare cost, accuracy, latency. Make the architecture decision with data.

2. Implement frame sampling for a video task. Take a 5-minute video. Try 1 fps, 0.25 fps, scene-detection-based sampling, and audio-first-then-frames. Compare cost and quality of the resulting analysis.

3. Set up a code agent against a real (small) project. Use Claude Code or build a minimal version with the tool stack. Pick 5 GitHub issues from your own backlog. Measure: how many resolved? Cost per resolution? Where did it get stuck?

4. Build a private SWE-bench-style eval. 20 cases from your own codebase or historical bug fixes. Run two code agents (or two prompt variants) and compare. This is your Week 8 discipline applied to code.

5. Read three real code-agent traces. Find published Claude Code or Cursor session traces (or run your own). Categorize the failure modes. What separates good runs from wandering ones?

Read this

• The Claude Computer Use technical report. Anthropic's writeup of agentic browser interaction.
• SWE-bench paper (Jimenez et al., 2023) and the SWE-bench Verified update. The standard reference for code agent eval.
• "Building Effective Agents" (Anthropic, 2024-25). Foundational pattern reference; mostly applies to code agents too.
• Cursor and Windsurf engineering blogs. Production code-agent architecture writeups.
• Mistral OCR documentation. The 2025-26 leader in dedicated OCR.
• Gemini and OpenAI multimodal docs. Reference for the audio/video API surfaces.
• Eugene Yan's writeups on document understanding. Practical, opinionated, useful.

Interview prompts

1. Walk through how you'd design a production document understanding pipeline at 1M docs/month.
2. When does the cost equation favor pure-VLM vs OCR-then-LLM?
3. Why is video so much more expensive than images, and what are the optimization levers?
4. Describe the canonical code-agent loop. What tools are essential?
5. What's the role of the sandbox in code agents, and what happens without one?
6. How would you read a published SWE-bench score? What does it tell you, what doesn't it?
7. What separates a production code agent from a benchmark-passing one?
8. How do you manage context in long-running code agent sessions?
9. When should you use audio-direct models vs Whisper-then-LLM?
10. What's the trust progression for deploying a code agent — and where would you stop?

What "done" looks like

By the end of this week you should be able to:

• Choose VLM vs OCR-then-LLM architectures based on volume, quality requirements, and cost.
• Estimate the cost step-up of adding multimodal to an existing system.
• Apply frame sampling, audio-first, and resolution tuning to make video workloads viable.
• Design and deploy a code agent with appropriate sandbox, tools, and trust boundaries.
• Read SWE-bench (and similar) scores critically and build private evals for your specific use case.
• Make the engineering tradeoffs between pure VLM, tier-routed pipelines, and specialized models.

If you can do those, you're ready for Week 11 — symbolic reasoning and reliability — where we cover the techniques (verifiers, tool use, retrieval-grounding, formal methods integrations) that make LLM systems reliable enough for high-stakes deployment. Multimodal and code agents both need these techniques; the next two weeks make them rigorous.