Interactive · Week 10 · Section 6

Two agents, same task, very different traces.

Both agents are asked to fix the same Python bug. Agent A is well-prompted with clean tool design; Agent B has weak tool docs and no context-management discipline. Both succeed in the end — but A takes 8 steps and $0.12, while B takes 20 steps and $0.55. Click any step to see the agent's reasoning. The visceral lesson: the difference between a code agent that ships in production and one that doesn't isn't model capability, it's tool design and prompt engineering. Same model can produce both traces.

Task Python · pytest SWE-bench-style

Fix the failing test in test_string_utils.py

The function truncate_string(s, n) in string_utils.py should truncate a string to n characters and append '...' only if truncation occurred. The test test_no_truncation_short_string is failing because '...' is being appended even when no truncation happened.

def truncate_string(s, n): return s[:n] + '...' # bug: always appends '...' # Failing test: def test_no_truncation_short_string(): assert truncate_string('hello', 10) == 'hello' # currently returns 'hello...'

Model: Claude Sonnet 4.6 Both agents have the same model, same task What differs: tool design + system prompt

Total steps

8 vs 20

B uses 2.5× more steps

Tokens

30k vs 141k

B uses 4.7× more tokens

Cost

$0.10 vs $0.47

B costs 4.7× more

Wall time

~45s vs ~3m

B takes 4× longer

Outcome

PASS vs PASS*

*after recovery

Side-by-side trace · click any step to expand

Hide wasted/wrong steps

Agent A · Efficient

Read · understand · fix · verify · done.

Tight tool design · senior-engineer system prompt · context-managed

Agent B · Wandering

Explore · explore · misfix · recover · finally fix.

Loose tools · weak prompt · accumulating context · same model as A

What separates Agent A from Agent B

Tool design. Agent A has a typed read_file(path) + edit_file(path, old, new) + bash(cmd) stack with clean error messages. Agent B has the same tools but with overly broad bash defaults and verbose error output, which inflates the context and confuses the agent.
System prompt. Agent A is told: "always run targeted tests with -x; read before editing; verify after every change; don't read files you don't need." Agent B has a generic "you're a coding agent" prompt and reaches for any file that might be relevant.
Same model, same task. Sonnet 4.6 in both cases. The 4.6× cost difference is purely engineering, not capability. Production code agents that ship are the ones with this engineering done well.
Context accumulation is the killer. Each step adds tokens to the conversation history. Agent B's later steps process 8000+ token context (vs A's 4000). Agent B's wrong-path edits (steps 10, 13) and recovery (step 17) each cost 5× more than Agent A's targeted single edit because the context has bloated.
Production budget reality. At 1M tasks/year, the difference is $100k (Agent A) vs $470k (Agent B). The "$0.37/task savings" is the engineering ROI of building the agent right vs shipping the prototype.
Trust progression matters. Agent B made wrong edits (steps 10, 13) that had to be reverted. In a production system without human-in-the-loop review, this would have shipped buggy code. The "agent runs in branch + human approves PR" pattern catches this; "agent commits to main" doesn't.

Both agents use Claude Sonnet 4.6 ($3/M input, $15/M output). Per-step costs reflect actual conversation token usage (mostly system prompt + accumulated file contents from prior steps). Agent A: ~30,000 total tokens summing to $0.10/task. Agent B: ~141,000 total tokens summing to $0.47/task — a 4.65× cost ratio. Wall time estimates assume ~5 second average per tool call (including LLM call + tool execution). Step kinds are color-coded: productive · verification · wasted · wrong path · recovery · final. Source-of-truth lives as <CodeAgentTraceComparator/> in the curriculum codebase.