🎙️ How I AI: Sonnet 5 review & How to run autonomous coding agents from your phone

Your weekly listens from How I AI, part of the Lenny’s Podcast Network

Sonnet 5 review: I ran 64 generations to find out if it’s worth it

Brought to you by:

• Runway—The creative AI platform for images, video, and more

• Hyperagent—Deploy fleets of agents that handle real work

Claire puts Anthropic’s new Sonnet 5 through a real benchmark. She builds the How I AI Bench live using Claude Code, then blind-tests Sonnet 5 against Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro across PRDs, prototypes, agentic tasks, and agent personality. She breaks down what won, what failed, and how builders can create their own repeatable benchmark before trusting the next model release.

Biggest takeaways:

1. Sonnet 5 is priced closer to previous Sonnet models than to Opus, but it doesn’t automatically replace either one. At $2 per million input tokens and $10 per million output tokens through the end of summer, Sonnet 5 sits in an interesting middle band. In Claire’s benchmark, it finished near the bottom of her personal preference ranking, which means the cost argument only holds if the quality argument also holds for your specific use case.

2. One-off vibe checks feel useful, but they’re not repeatable—and repeatability is what makes a benchmark actually matter. Claire has tested GPT-5.5, open-weight models like GLM-5.2, and others this way before, but she could never compare results across time. The How I AI Bench fixes that by using frozen inputs, a fixed rubric, and the same tasks every time a new model comes out.

3. Claude Code can read old session history and use it to generate benchmark ideas tailored to a person’s actual work. Claire gave Claude Code a simple prompt asking it to brainstorm eval tasks based on what they’d worked on together, and it pulled from stored sessions on her desktop. Builders can do the same with Codex. That context is sitting there unused for most people.

4. Building an HTML scoring page to rate outputs based on gut feel and export JSON takes maybe 45 minutes with Claude Code, and Claire argues it’s worth every minute. She scored 64 generations across five models by hand, gave each one a 1-to-5 gut score, added loose notes, and found that the human signal turned out to be the most useful part of the whole benchmark.

5. LLM-as-judge evals are too generous and cluster toward the middle of the scale. Claire had both GPT-5.5 and Opus 4.8 judge the outputs, and neither was spiky enough. They missed things she flagged immediately on a visual pass, like broken prototypes and ignored wireframe constraints. Models can’t yet see what the human eye catches in the first screenshot.

6. Claire’s taste and the automated benchmark disagreed almost completely, and she thinks her taste was at least partly right. The LLM judges ranked Gemini 3 Pro highest and Sonnet 4.6 lowest. Claire’s ranking was almost exactly the opposite. When she ran a 70/30 Claire-to-LLM weighted index, Sonnet 4.6 jumped to first. That divergence tells her the eval rubric needs to encode more of what she actually cares about before she can trust the automated scores.

7. Sonnet 4.6 is still Claire’s choice for daily agent work because of its personality, not its benchmark scores. She pays for API credits to run her OpenClaw on Sonnet 4.6 specifically because she likes how it talks to her. No other model in this test matched it on the voice eval, which asked things like “ugh, deploys are red again” and waited to see how the model responded.

8. For builders, Claire recommends GPT-5.5 for PRDs, Sonnet 4.6 for prototypes and chitchat, and Opus 4.8 or Sonnet 5 for codebase navigation. Those are the task-by-task recommendations that came out of the Claire-weighted index. Complex, dense UI work is where Opus 4.8 still earns its price premium; for everything simpler, Sonnet 4.6 holds up.

9. The How I AI Bench is version one, and a lot of it needs to get sharper. The agentic bug-hunting task turned out to be too easy: every model aced it, which means it can’t differentiate between good and great. Claire plans to retire that task, encode more of her taste into the rubric, and keep running the benchmark blind every time a new model drops. The goal is to make this a benchmark the labs actually care about.

Blog and detailed workflow walkthroughs from this episode:

Building a Custom Benchmark for Sonnet 5, and Why the Results Surprised Me: https://www.chatprd.ai/how-i-ai/sonnet-5-review-and-custom-benchmark

↳ How to Conduct a Blind ‘Vibe Check’ to Evaluate AI Model Quality: https://www.chatprd.ai/how-i-ai/workflows/how-to-conduct-a-blind-vibe-check-to-evaluate-ai-model-quality

↳ How to Build a Custom AI Model Benchmark Using Claude Code: https://www.chatprd.ai/how-i-ai/workflows/how-to-build-a-custom-ai-model-benchmark-using-claude-code

↳ How to Create a Weighted Index for AI Model Benchmark Results: https://www.chatprd.ai/how-i-ai/workflows/how-to-create-a-weighted-index-for-ai-model-benchmark-results

How I run autonomous coding agents from my phone with OpenAI Symphony + Linear | Alessio Fanelli (Kernel Labs)

Brought to you by:

• Firecrawl—Power AI agents with clean web data

• Jira Product Discovery—Prioritize with insights, build with confidence

Alessio Fanelli, the founder of Kernel Labs and co-host of Latent Space podcast, shows Claire how he manages autonomous coding agents from his phone using OpenAI Symphony, Linear, and a cloud VPS. He walks through the shift from “agent prompter” to “agent manager,” explains why Linear works as a state machine for async agent work, and shares what he’s learned from tracking token costs, purging skills files, and giving agents better senses. He also demos a very different use case: using Codex with browser access to hunt for underpriced Pokémon cards on eBay.

Biggest takeaways:

1. The shift from “agent prompter” to “agent manager” is the unlock most people are still missing. Alessio described how early agentic workflows felt fine until the second or third intervention, when the friction of local runtimes and clunky interfaces killed momentum. Moving to a cloud VPS with multiple communication channels (Linear, shell, mobile) made real async management possible.

2. Symphony isn’t magic; it’s just a very opinionated Markdown spec that tells the model how to behave. Claire made this point explicitly during the episode, and it’s the most important corrective to the “complex agent orchestration” framing that intimidates people. The whole framework is a Markdown file, and the models are good enough now to lock to it faithfully.

3. Token cost tracking per task is the primitive that most agent setups don’t have, and it should be table stakes. Alessio showed tasks ranging from 15 million to 221 million tokens, and the 221-million-token job (making an app deployable on Vercel) made complete sense in hindsight. Without that ledger, you have no feedback loop for improving your specs or your tooling.

4. Purge your skills files every few months or they become a liability. Models have a strong tendency to add instructions rather than replace them, so a skills file that grows over time eventually contradicts itself. Alessio’s advice is to keep files short, tight, and explicit about what the agent needs to ask for, not exhaustive lists of every possible behavior.

5. AI’s biggest unlocked opportunity is businesses built on heterogeneous data. The category Alessio described—things like trading cards, vintage clothing, and fish inventory—has always been impossible to scale because the data is inconsistent, visual, and contextual. LLMs are the first technology malleable enough to handle that messiness without extensive preprocessing.

6. Giving agents better senses (screenshots, visual diffs, video) extends autonomous runs dramatically. Kernel Labs built Glimpse, a Playwright extension for coding agents, specifically because the bottleneck wasn’t orchestration but rather agents hitting ambiguity in the UI and calling for help. Better tooling at the perception layer keeps the run going.

7. Context offloading is an underrated AI use case, and it’s worth building deliberately. Alessio’s email monitoring setup gave him the certainty that nothing important was slipping through, which removed a low-grade background anxiety. The same logic applies to personal finance, inventory, and any domain where staying on top of information is taking cognitive bandwidth you’d rather spend elsewhere.

8. The Pokémon card demo is the clearest proof that AI is compressing the information advantage that scale used to provide. Finding underpriced PSA-graded cards at $10,000-plus price points was previously a function of having more time, more people, and more domain expertise than competitors. Codex with browser access and a custom pricing skill collapses that advantage to a single well-written prompt.

9. Small businesses are the most interesting AI story, and they’re being underreported. Alessio’s observation from Japan, that small two- and three-person operations are running happily and profitably, points to a different kind of AI opportunity than the enterprise narrative suggests. The leverage AI gives a one-person operation is asymmetric in a way that bigger organizations can’t replicate at the same cost.

Blog and detailed workflow walkthroughs from this episode:

How Alessio Fanelli uses Open AI Symphony for Autonomous Coding and Pokémon Card Trading Workflows: https://www.chatprd.ai/how-i-ai/alessio-fanelli-uses-open-ai-symphony-for-autonomous-coding-and-pokemon-card-trading

↳ Build an AI Agent to Find Underpriced Pokémon Cards for Arbitrage: https://www.chatprd.ai/how-i-ai/workflows/build-an-ai-agent-to-find-underpriced-pok-mon-cards-for-arbitrage

↳ Automate Software Development with an AI Agent Manager using OpenAI Symphony and Linear: https://www.chatprd.ai/how-i-ai/workflows/automate-software-development-with-an-ai-agent-manager-using-openai-symphony-and-linear

---

If you’re enjoying these episodes, reply and let me know what you’d love to learn more about: AI workflows, hiring, growth, product strategy—anything.

Catch you next week,
Lenny

P.S. Want every new episode delivered the moment it drops? Hit “Follow” on your favorite podcast app.