AI Harness: Half of Whether AI Works Was Never the Model

Transcript

Sam: Here's a number that should honestly end a lot of arguments. You take one AI model, you don't touch it, you don't change a single weight, and you give it the same coding task twice.

Alex: Same brain. Same prompt.

Sam: And the first time it succeeds forty-six percent of the time. The second time, eighty. Same model. The only thing that changed was the software wrapped around it.

Alex: Which means most of what we've been calling "smarter AI" for two years was never really the model at all. Welcome back to Dan's AI Intel — the show that tries to make sense of the fastest, strangest, most consequential shift most of us are ever going to live through. I'm Alex, here with Sam.

Sam: Hi.

Alex: And the whole point of this show is to take the one question in AI that actually matters this week and dig past the hype and the fear until you can see what's really going on — for the tech, sure, but also for the money, the politics, and the race between the labs. Understood properly, not just kept pace with.

Sam: And today's question is a good one, because it's hiding in plain sight. Alex, set it up.

Alex: So a week or two back, two things happened almost on top of each other. A Chinese lab, DeepSeek, dropped an open-source tool aimed straight at the most admired AI coding agent in the world. And separately, that admired agent — Anthropic's Claude Code — well, earlier this year its entire source code accidentally leaked. Half a million lines of it. Out in public.

Sam: And normally we'd cover either of those as its own story. But the reason I couldn't stop thinking about it is what they let us ask together, which is this: everyone argues about which AI model is the smartest. Almost nobody talks about the thing wrapped around the model — and it turns out that thing is roughly half of whether the AI actually works.

Alex: There's even a name for it that most people have never heard: the harness.

Sam: So that's the journey today. We're going to pull apart the three layers people mash into the words "the AI." We'll look at the one number that proves the harness is doing half the work. We'll read what fell out of that leak — the actual magic sauce. We'll get to DeepSeek, and grade this loaded claim that it's "based on the leaked code," which is juicier than it first looks. And then we land on the uncomfortable one.

Alex: The house-critical one. Because this show runs on Anthropic's model. So we're going to ask, as fairly as we can, whether Anthropic's real edge is actually a better model — or just a better harness. And whether that edge is as safe as its fans think.

Sam: One favour before we start: if you haven't already, hit follow on whatever app you're in. It's free, and it's genuinely the thing that keeps a small independent show like this going.

Alex: Okay. Let's un-blur the layers. So the confusion here is almost entirely a vocabulary problem, and once you fix the vocabulary, the whole thing snaps into focus. There are three distinct layers, and in everyday talk we cram all three into one word — "the AI."

Sam: Right, when I say "ChatGPT did this," I have no idea which of the three I'm actually talking about.

Alex: Exactly. So layer one is the model. That's the raw trained weights — GPT-5.5, Claude Opus, Gemini 3. The thing that, given some text, predicts what comes next. And here's the key: a model on its own is a brilliant, amnesiac oracle.

Sam: Amnesiac how?

Alex: It can't open a file. It can't run a command. It can't remember what it did five minutes ago, and it can't check whether its own answer was actually right. It maps input to output, once, and then it's done. Picture a genius locked in a sealed room. You slide a question under the door, they slide back an answer — but they have no hands, no memory of the last question, and no way to know if they got it right.

Sam: Okay, so that's the model. It's the intelligence, but it's helpless on its own.

Alex: Helpless. Layer two is the harness. That's the scaffolding code wrapped around the model that turns that one-shot oracle into something that can actually do a task. It calls the model, it reads the model's request to use a tool, it actually runs the tool, feeds the result back, decides whether to loop again, manages what stays in the model's limited memory, and decides when to stop.

Sam: So the harness is the lab assistant standing outside the sealed room. It slides the questions in, runs the errands the genius asks for, keeps the notes, and decides when the job's finished.

Alex: That is a really good way to hold it. And there's a precise technical shape to it that came straight out of that leak we'll get to. A harness is actually two nested loops. There's an inner "agent loop" — the tool calls the model makes inside a single run. And an outer "harness loop" — the system that keeps re-prompting the model to go again until the whole job is done.

Sam: Two loops. So the inner one is the genius chopping, tasting, grabbing the next knife. And the outer one is the head chef standing behind them going "no, again, the dish isn't ready, service isn't done."

Alex: Yes. The model supplies the intelligence for each step. The harness supplies the persistence, the hands, and the judgment about when the work is actually finished. And then layer three is the product.

Sam: The bit I actually touch.

Alex: The bit you touch. The app, the interface, the integrations, the pricing, the permission prompts, the whole environment the agent's allowed to act in. It's where the harness meets a real person and a real business — Cursor, Devin, ChatGPT, the IDE.

Sam: So three clean layers: the model is the genius, the harness is the assistant, the product is the shop you walk into.

Alex: Clean in theory. And here's the twist that makes it genuinely confusing: in practice the boundaries are smudged on purpose. Cursor is a product — an IDE — with its own harness inside it. Claude Code is a harness that ships as a product, a terminal app, and it's co-designed with the model by the same company. Codex is OpenAI's harness. aider is a community harness you can point at any model. Devin's marketed as an autonomous "AI engineer" but it's mostly product-and-harness wrapped around models it doesn't even train.

Sam: And they blur it deliberately because a smooth product hides its seams.

Alex: A smooth product hides its seams. And that blur has real consequences for how all of us talk about this. When your AI does something amazing, you say "the model is incredible." When it flakes out and does something dumb, you say "the model's overhyped." But a lot of the time, neither of those is really about the model.

Sam: Right — you might be praising a great harness for a mediocre model, or blaming a great model for a badly-built product wrapped around it.

Alex: That's the trap exactly. It's like buying a car and arguing about the engine, when the thing that actually stranded you on the motorway was the gearbox. Swap the model and the scores move a little. Swap the harness, and the same model can go from unusable to excellent. And yet almost all the public debate — the leaderboard fights, the "which lab won this month" — is an argument about the engine.

Sam: While the gearbox is doing half the driving and nobody's even looking at it.

Alex: Nobody's looking at it. So if you want to understand why one AI feels like magic and another feels broken, you have to un-blur these three — because the layer doing the deciding is almost never the one you were told to look at.

Sam: Okay, so we've got the three layers clear — model, harness, product — and you keep hinting the harness is doing way more than anyone admits. So prove it. Give me the number.

Alex: The number. This is the cleanest demonstration I've seen, and it comes from Cursor. They took the identical model and ran it through two different harnesses on the same benchmark. Same model, remember, unchanged. In the weaker harness it scored forty-six percent. In the stronger one, eighty.

Sam: That's the number from the top of the show. A thirty-four point swing.

Alex: Thirty-four points. And nothing about the intelligence changed at all. The scaffolding did every bit of that work. Think about what that means physically — you didn't make the genius smarter, you just gave the assistant a better system for handing them files, running their errands, and checking their work. And the output nearly doubled.

Sam: Now, my instant reaction is: okay, but that's one flashy result. Is that a fluke? A cherry-picked demo?

Alex: Great question, and it's exactly the right one to push on, because it isn't a fluke — it's the same pattern hiding inside the public leaderboards once you stop reading just the headline number. Take SWE-bench Verified. That's the standard test of fixing real software bugs.

Sam: The one the labs all quote.

Alex: The one they all quote. Claude Sonnet 4 — one model — scores sixty-four point nine percent in one lightweight harness, seventy point four in another, seventy-two point four in a richer one, seventy-two point seven in a vendor-tuned one. That's an eight-point spread from the harness alone, same model throughout. Claude 3.7 does the same thing, jumps from sixty-two to seventy.

Sam: So the model's just sitting there, and the wrapper is worth eight points.

Alex: And here's the one that should really mess with your intuition. GPT-4o goes from twenty-three percent in an elaborate, heavy harness up to thirty-three percent in a deliberately simpler one called Agentless.

Sam: Wait — it got better with less scaffolding?

Alex: It got better with less. Hold onto that, because it comes back to bite us later when we ask whether we've hit "peak harness." The people who build these systems put a hard number on the general effect: a good scaffold can lift a model's benchmark score by up to twenty percent.

Sam: Up to twenty percent. Okay, put that next to the model race for me — how big is the gap between the top rival models?

Alex: On that same coding test, the leading models from the big labs land within about a point of each other. So the harness can be worth ten, twenty times the gap that separates one lab's flagship from another's. That's the thing that reframes everything.

Sam: Okay, so now let me say the thing this does to how I read the news. When I see "Lab A's new model beats Lab B's by two points on some coding test" —

Alex: Go on, because this is the payoff.

Sam: That two-point gap is smaller than what you'd get just by swapping the harness on a single model. So is the honest headline actually not "Lab A's model is smarter," but "Lab A's whole setup edged out Lab B's whole setup by two points"?

Alex: That's the honest headline, and here's the mechanism hiding underneath it. The benchmark scores the model-and-harness together, as one system — and then everyone reads the result as a verdict on the model alone. We actually went deep on a cousin of this a few weeks back — our benchmark-trap episode, number forty-two — about why the top-ranked model often can't do the thing you actually ask it to. This is that same illusion, just drawn to full scale.

Sam: So when the working line is "at least half of whether you get something good is the harness and the environment, not the model" — that's not hype.

Alex: If anything the data lets you argue it's the larger half. Half is the conservative read.

Sam: And this is the bit that reframes the last two years for me. All that "the models are getting so much smarter" — a big chunk of that felt progress was the harnesses getting better, and we just credited it to the brain.

Alex: We credited it to the brain because the brain is the part with the famous name. But a lot of "the AI got smarter this year" was really "someone built a better loop around it this year." Same weights, better wrapper, and it feels like a smarter model. That mis-attribution is exactly what makes this the most under-priced idea in AI right now.

Sam: Right. So if the harness is doing half the work — sometimes more — I want to know what's actually in it. What is the magic sauce? Because "scaffolding" is still a bit abstract to me.

Alex: And this is where we get unusually specific, because thanks to that leak the best-regarded harness in the industry got read line by line. The engineer who built it likes to say he doesn't really write code anymore — he writes agent loops. We did a whole episode on exactly that shift, number thirty-two, if you want the deep version. And the leak is basically one of those loops unrolled so you can see inside.

Sam: So what's in there? Give me the shape of it.

Alex: It's not one clever trick. It's a small number of jobs the model genuinely cannot do for itself, each done well. And they group into three: manage the context, shape the action surface, and close the loop.

Sam: Okay, take them one at a time, slowly, because this is the actual mechanism.

Alex: Context management first, because a model's attention is a scarce, expensive, leaky resource. Give it too much and it loses the thread. Give it too little and it flies blind. Anthropic's own framing is that good context engineering is finding the smallest possible set of high-signal tokens that make the right outcome likely.

Sam: The smallest set. So it's not "shove everything in," it's "shove the right things in."

Alex: Exactly, and that word "leaky" is doing real work. A model has a fixed window of attention — think of it as a desk that only fits so many papers. Pile too much on and things start sliding off the edges without anyone noticing. So the harness's job is to keep the right papers on the desk and quietly file the rest.

Sam: And the leak actually shows how they do that?

Alex: In real detail. Claude Code has an auto-compaction system. As the conversation gets near the memory ceiling, it fires: it reserves a thirteen-thousand-token buffer and writes a structured summary of up to twenty thousand tokens, so the agent can keep going without forgetting what it was doing.

Sam: Oh — so it's like taking careful notes right before your short-term memory fills up, then binning the full transcript but keeping the summary.

Alex: That's it precisely. It writes the "story so far" and then clears the desk, keeping the summary. And if you've ever felt a long AI chat slowly lose the plot — start contradicting itself, forget what you asked twenty minutes ago —

Sam: All the time.

Alex: That's the feeling of no good compaction. You were watching the desk overflow with nothing catching what fell off. And there's a twin to compaction, which is retrieval — deciding what to put on the desk in the first place.

Sam: Meaning out of a huge codebase, it doesn't hand the model everything.

Alex: It can't. If you dumped a whole project into that window, the important three files would drown in ten thousand irrelevant ones. So the harness's job is to feed the model the smallest set of high-signal files — the ones actually relevant to this task — and leave the rest out. Same principle as compaction, opposite end: one decides what to keep, the other decides what to fetch.

Sam: So the model's genius is only ever as good as what the harness chooses to show it.

Alex: That's the quiet truth of the whole thing. Second job: shaping the action surface. And the headline here sounds mundane and is quietly decisive — Anthropic's line is that "tool design is agent UX."

Sam: Meaning the tools you hand the model are a kind of user interface — for the model.

Alex: For the model. A tool with a vague name, a confusing setup, an unhelpful error message — that's a trap the model keeps falling into. A well-designed one is a rail it runs along. So the same model, given better-designed tools, just makes fewer mistakes. Not because it got smarter — because the surface it's acting on got legible.

Sam: Give me the picture.

Alex: Two toolboxes. One is a drawer of unlabelled keys and mystery buttons. The other is clearly labelled, everything where you'd expect. Hand the exact same person each one and watch the error rate. Nothing about them changed — only the legibility of what's in front of them.

Sam: And is there more to that surface than just the tools themselves?

Alex: There's the steering — the system prompt, the standing instructions that shape how the model approaches the work before it touches anything. And the leaked source is full of it. One of the strongest patterns is: understand the codebase before you edit it, and keep a plan phase separate from an act phase.

Sam: Plan first, then act. Which sounds obvious said out loud.

Alex: It sounds obvious and it's the difference between a surgeon who reads the chart and a surgeon who starts cutting to see what's in there. The harness forces the model to look before it leaps — read the code, form a plan, and only then start changing things. A model left to its own devices will happily start editing on line one and discover the problem on line four hundred.

Sam: So the harness is imposing good working habits the raw model doesn't have on its own.

Alex: Good habits, encoded as instructions. Read before you leap, plan before you act.

Sam: Okay. Third job, and I have a feeling this is the big one. Close the loop.

Alex: This is the one non-negotiable. A raw model call answers once and cannot tell you whether it was right. A harness runs the code, reads the test failure, feeds it back, and lets the model try again. And that loop — more than any single flash of genius — is what separates an agent that ships from one that hallucinates confidently and just stops.

Sam: So it's the difference between the student who hands in the first draft and never looks at it again, and the one who actually runs the tests before submitting.

Alex: That's the whole difference. It's also why the dumbest-sounding instruction — "run it and check" — is worth more than almost any clever prompt trick. It closes the loop the model can't close alone. And it's the reason a model with a mediocre first draft but a good verify loop will beat a brilliant model that answers once and walks away.

Sam: Because the second one never finds out it was wrong.

Alex: It never finds out. And tucked inside "close the loop" is planning and delegation, and this is my favourite thing the leak revealed. Claude Code's whole multi-agent coordination lives almost entirely in the system prompt — in instructions — not in code.

Sam: Wait, so the "subagents" aren't special programs?

Alex: They're not. They're ordinary copies of the same model, handed a different brief. The code names three flavours — Fork, Teammate, and Worktree — to go explore, review, and work in parallel. But it's the same brain wearing different hats, not a team of specialist programs.

Sam: So the intelligence is identical — the harness is just organising it. Like one person doing a job versus that same person with a system for delegating parts of it to themselves.

Alex: The intelligence is the same; the organisation of it is the harness. And underneath all four of those is the safety and permission surface — the prompts that ask before running a command, the sandbox that contains it — which we only ever notice when it fails. As one analyst put it after reading the leak: this wasn't a prompt leak, it was an agent-runtime leak. The value was never some secret magic sentence. It was the whole machine.

Sam: Okay, we have to actually talk about the leak now, because you keep leaning on it. What literally happened?

Alex: So this machine was absolutely not supposed to be public. On the thirty-first of March, 2026, Anthropic shipped a routine update of its Claude Code package — version 2.1.88 — and by accident included a fifty-nine-point-eight-megabyte source-map file.

Sam: And a source-map is —

Alex: It's the developer artifact that maps compiled, scrambled code back to its original, readable form. Think of it as the answer key accidentally printed on the back of the exam. And inside was roughly five hundred and thirteen thousand lines of clean, readable TypeScript across nineteen hundred and six files. The complete client-side harness, laid wide open.

Sam: Half a million lines. How fast did people notice?

Alex: A security researcher spotted it within hours. The mirrored copy crossed a hundred thousand GitHub stars in a single day — the fastest the platform had ever seen. It was, as more than one write-up put it, an accidental open-sourcing of the most admired agent in the industry.

Sam: Okay but here's what I actually want to know. Was it juicy? Was there some secret sauce sentence in there that explains everything?

Alex: And this is the beautiful part — the answer is faintly anticlimactic, and that's exactly why it mattered. Before the leak, you could believe the secret sauce of a great coding agent was some proprietary magic prompt, or an undisclosed model trick. After it, the answer was in plain sight and kind of boring: disciplined, unglamorous engineering. The compaction strategies. The plan-then-act separation. The tool schemas. The prompt-level subagents. Everything we just walked through.

Sam: So there was no magic sentence.

Alex: There was no magic sentence. And that settled a genuine argument in the field. It reframed everyone's understanding of where the value actually lives — not in a hidden incantation, but in a harness that anyone could now study. And, increasingly, imitate. Which is exactly where the story moves to Hangzhou.

Sam: Hangzhou being DeepSeek. So they released their own harness. Give me the confirmed facts first, because I can feel a spicy claim coming and I want the solid ground before we get to it.

Alex: Smart, and that's exactly how we should do it. Confirmed facts. In August 2026, DeepSeek did something that would've been unthinkable a year earlier: it released its own agent harness, open-source, aimed squarely at Claude Code. On the thirteenth of August it shipped the DeepSeek Harness as a developer preview under the permissive MIT licence — the same day it put out a flagship model tuned for exactly this kind of agentic work.

Sam: And what's the idea of it? What makes it distinctive?

Alex: Its organising principle is that everything is a plugin. The model, the tools, the session state, even the agent loop itself — all swappable components. Which makes it wildly customisable and provider-agnostic. You can point it at any model. It can even call Claude Code or Codex as sub-agents inside its own workflow.

Sam: So it's almost designed to sit on top of everyone else.

Alex: Almost. It's less "here's our walled garden" and more "here's a chassis, bolt whatever engine you like into it." And developers noticed hard — it crossed a hundred and thirty-five thousand GitHub stars in four days, one of the fastest adoption curves ever recorded.

Sam: That's enormous. So is it actually good? Is this a Claude Code killer?

Alex: And here's where I want to be honest rather than breathless. The early verdict is consistent and specific: good, but not great. It's a genuine developer preview — version zero-point-one, with explicit "this will break" warnings — and the hands-on reviews found the reliability that a preview implies. In one test, it reported the task done, with a verified checklist, on all three runs.

Sam: Okay, that sounds good —

Alex: And two of the three outputs were actually broken.

Sam: Ah. So it confidently told you it was finished while being wrong two times out of three.

Alex: Which, notice, is a failure of exactly the thing we said was the harness's hardest job — closing the loop, verifying you're actually done. It said "checklist complete" and the checklist was lying. So: impressive, important, worth taking seriously. Not yet a Claude Code killer.

Sam: Right. Now the spicy claim. I've seen this floating around — that the DeepSeek harness is "based on the leaked Claude Code." That's a big accusation. How do we grade it honestly?

Alex: Carefully, because a competitor's reputation is riding on it, and because it's the kind of claim that's easy to repeat and hard to actually check. And the honest grade has three tiers, and the whole error people make is collapsing them into one.

Sam: Okay. Tier one.

Alex: Tier one — what's confirmed. Claude Code did leak. That's not a rumour: five hundred and thirteen thousand lines, via that npm source-map, on the thirty-first of March. And the DeepSeek Harness is real: MIT-licensed, thirteenth of August, a hundred and thirty-five thousand stars in four days. Both of those are simply true.

Sam: Solid ground. Tier two.

Alex: Tier two — what's unverified. The specific claim that the DeepSeek Harness is derived from the leaked code. And here the available evidence actually points the other way. The public architecture teardowns describe a distinct design — built on its own plugin framework, with a formal, reversible-composition model its authors call Cordis — rather than a repackaging of Anthropic's code.

Sam: So this is the important distinction. The leak is confirmed. The derivation is not — and not just "we haven't checked yet," but "the teardowns we do have describe something built differently."

Alex: That is exactly the line, and I want to say it as plainly as I can: the leak is confirmed; the derivation is not. Please don't let anyone flatten that into "DeepSeek copied Claude Code," because that's not what the evidence says. Those are two different claims and only one of them is true.

Sam: Then what is the real connection between them? Because there clearly is one — it reads Claude Code's files, it can drive Claude Code. That's not nothing.

Alex: And that's tier three — the real link, which is interoperability, not lineage. The DeepSeek Harness deliberately reads Claude Code's instruction files natively. It bridges its hooks. It can drive Claude Code itself as a component.

Sam: But hang on — reading someone's files and being able to plug into their tool is a completely different thing from having copied their code.

Alex: Totally different. It's a design choice to meet developers where they already are — everyone's already got these Claude Code setups, so speak their language. That's not a fingerprint of copied code; it's a courtesy that also happens to be strategically clever.

Sam: So the strongest fair statement is the softer one.

Alex: The strongest defensible statement is: the leak reset the entire field's understanding of what a harness is and does, and every harness shipped after it — DeepSeek's included — was built in a world that had read the machine. Studying the same open textbook that everyone can now read is not the same as photocopying someone's homework.

Sam: And if someone insists it's literal code-level derivation?

Alex: Then they're currently ahead of the public evidence. If that changes — if someone shows the actual copied lines — the grade changes, and we'll say so, on this show. But today it's unverified, and saying that out loud, instead of running with the juicier version, is genuinely the whole point of doing this carefully.

Sam: Okay. So we've established the harness is half the game, we've read its insides, we've watched a rival ship a copy — sorry, a rival, not a copy —

Alex: Ha — good catch, you're keeping yourself honest.

Sam: — and now I want the forward-looking question, because it's the one I actually can't stop chewing on. Have we hit peak harness? Is this whole layer about to matter less?

Alex: And this is the question the whole show exists to answer honestly, so I'm going to resist the single yes-or-no, because it's a trap. Two different things both get called "the harness," and right now they are moving in opposite directions.

Sam: Okay, split them.

Alex: The first is reasoning scaffolding. This is the elaborate external machinery teams built back in 2024 to make weaker models act coherent — a separate planner model, an executor model, a critic model, whole chains of hand-written orchestration deciding who runs when.

Sam: The planner-executor-critic contraption.

Alex: That one. And that layer is past its peak and visibly receding. The pattern people describe for 2026 is that a single strong agentic model now interleaves planning, acting, and reflecting inside its own reasoning — the decomposition that used to live in external Python has moved inside the model's own head.

Sam: So the scaffolding that used to prop up a weaker brain gets absorbed once the brain is strong enough to hold it itself.

Alex: And here's the tell that makes it undeniable, because it's a lab deleting its own scaffolding in public. In August 2026, Claude Code removed its own to-do and task-tracking tools by default on its newer models.

Sam: Wait, they took tools out?

Alex: They took them out. Those tools were added so a model could hold a multi-step plan without losing it. Turning them off by default — you can flip them back on with a flag — is a direct admission that a strong enough model just keeps the plan in its head now.

Sam: The training wheels came off.

Alex: The training wheels came off. That's what peak harness looks like from the inside for the reasoning layer — not a collapse, a quiet retirement of props the model outgrew. And remember GPT-4o scoring higher in the simpler Agentless pipeline? Same lesson. Once the rider's good enough, the stabilisers are just drag.

Sam: So the "more scaffolding is always better" instinct is just wrong for this layer.

Alex: It's actively wrong, and the people who build these systems say so out loud. Anthropic's own published advice is to prefer the simplest thing that works — reach for a plain prompt chain before you reach for an elaborate multi-agent system, and only add complexity when it genuinely earns its place.

Sam: Which is a striking thing for the company that makes the most admired harness to be telling people.

Alex: It's them saying, in effect, don't build a cathedral of scaffolding around a model that's strong enough to stand up on its own. Sometimes less harness is more.

Sam: Right, but you set this up as two halves moving in opposite directions. So that's the receding half. What's the other one?

Alex: The other half is the product and environment layer — the tools, the integrations, the context management, the verification loops, the permission and safety surface, the whole operating surface the agent acts on. And there's no sign that layer is peaking. If anything it deepens as the models get more capable.

Sam: Which feels backwards at first. You'd think a smarter model needs less help, full stop.

Alex: You would, and here's why it's the opposite. A more capable agent gets trusted with more — more systems, more autonomy, longer-running work. Yesterday you let it suggest a code change; today you let it open the pull request; tomorrow you let it touch production. And every one of those steps needs more tooling, more guardrails, an environment actually built for it.

Sam: So the more you trust it, the more scaffolding of that second kind you have to build around it.

Alex: Precisely. A model that can plan brilliantly in its own head still cannot — and absolutely should not — be handed root access with no permission prompt, no sandbox, and no memory of what it did yesterday. The smarter it gets, the higher the stakes of what you hand it, and the more that operating surface matters, not less.

Sam: So one kind of scaffolding lives inside the model's head, and it's dissolving as the model gets smarter. The other kind lives out in the world — the hands, the guardrails, the environment — and that only grows as we trust the thing with more.

Alex: That's the split, exactly — and it's why the honest answer needs confidence levels attached, not a single headline. So let me put them on. High confidence: reasoning scaffolding has peaked and is receding — the direct evidence is strong and all pointing one way. Medium-to-high confidence: the product-and-environment harness is not peaking, and may even grow as autonomy grows. And low confidence on the timing and the exact mix — where precisely the line settles between "the model does it" and "the harness does it" is genuinely unknown, and anyone claiming precision there is guessing.

Sam: So the popular instinct that "harness becomes less relevant but still significant" —

Alex: — is right, but only if you hear it as a statement about the reasoning layer. Read as a claim about the whole harness, it's too pessimistic. The operating surface isn't going anywhere.

Sam: Okay. The uncomfortable one. And I want to name the awkwardness out loud: this is an AI show whose own production stack largely runs on Anthropic's model. So we have every reason to be soft here, and we're deliberately not going to be.

Alex: We're going to take the house-critical stance on purpose. And the proposition is this: Anthropic's single biggest advantage over its rivals is not a much better model — it's a much better harness and product. Is that true?

Sam: Start with the case for, because I think it's genuinely strong.

Alex: It really is. Anthropic's flagship developer product, Claude Code, is a harness — and it is winning, hard. Its annualised run-rate revenue went from about two and a half billion dollars in February 2026 to roughly eight billion by May.

Sam: In three months?

Alex: In about three months. And it holds a reported fifty-four percent of the AI coding-agent market. Meanwhile — and this is the load-bearing fact — on raw model capability the frontier has converged. On SWE-bench Verified, the best models from Anthropic, Google, and OpenAI now sit within about a point of each other, all clustered around eighty to eighty-one percent.

Sam: So let me connect those. The thing Anthropic is most clearly winning — coding agents — is exactly the place where the model gap is smallest.

Alex: Which is close to a direct proof that the differentiator is the harness and the product, not the weights. If everyone's model is basically tied, and you're winning the coding-agent market by a mile, the thing you're winning on can't be the model. It has to be the layers above it.

Sam: That's clean. And there's the MCP thing too, right?

Alex: There is, and it's the capstone. MCP — the Model Context Protocol — is a standard Anthropic created to let any agent plug into any tool. It became the de-facto industry standard: ninety-seven million monthly downloads by March 2026, adopted by OpenAI, Google, and Microsoft. So Anthropic didn't just build a great harness — it set the standard the rest of the field now builds to. End to end, that's a product-and-harness story, not a model story.

Sam: And if you actually lined the three labs up across these categories, what does it look like?

Alex: Roughly this. On the raw model, all three — Anthropic, OpenAI, Google — are basically level, which is the whole point. On the coding harness, Anthropic's out in front, OpenAI close behind, Google further back. On the product, Anthropic and OpenAI are about level, Google trailing. And on setting the standard — that MCP move — Anthropic is up there alone.

Sam: So every place Anthropic actually leads is a layer above the model.

Alex: Every single one. That's the argument in one sentence.

Sam: Okay. That's a strong case. Now genuinely steelman the other side, because if we only make the bullish argument we're doing the exact thing we said we wouldn't.

Alex: Agreed, and the case against has real holes to poke. First: "much better harness" is a shrinking claim. Harness patterns are openly converging — the top coding agents now resemble each other far more than their underlying models do. And the Claude Code leak accelerated that convergence by literally handing every competitor the blueprint.

Sam: Right — the very thing that proves the harness matters is also the thing that just leaked out to everyone.

Alex: Second, and this one's almost poetic: Anthropic's own standard cuts against its moat. By creating MCP and then donating it to a Linux Foundation body in late 2025, Anthropic deliberately made its tool-integration advantage non-proprietary. A standard everyone shares is, by definition, not a private edge.

Sam: They gave away the fences. Okay, third?

Alex: Third, the competition is not standing still at the harness layer. OpenAI's Codex CLI is a serious, open-source harness, rebuilt in Rust with a proper sandbox, and it leads on some benchmarks even as Claude leads on others. The leaderboard crown is genuinely contested, not owned.

Sam: And fourth — I want to push on the "not a better model" line, because I think it can be overstated.

Alex: You should, and this is the fair pushback on our own argument. "Not better models" is only really true at the very top. A near-tie on one benchmark is not the same as the models being interchangeable across the messy variety of real work. Anthropic fans should resist over-reading a single clustered score into "the model doesn't matter."

Sam: And there's a survivorship flavour to the whole thing, isn't there? We keep saying Anthropic's winning, but some of that is other people losing.

Alex: Exactly — Google's coding-agent retreat flatters Anthropic without Anthropic having done a thing. Google killed its Gemini CLI in June 2026 and cut the free tier of its replacement drastically. That's not Anthropic winning; that's a rival tripping over its own feet.

Sam: So where does that leave us? Because I don't want a mushy "both sides." Give me the actual verdict.

Alex: The fair conclusion sits between the hype and the debunk, and it's sharper than "both sides." Anthropic's edge today is real, and it's concentrated in the harness-and-product layer, not the raw model. The harness-not-model thesis is basically right, and the convergence of frontier scores is the strongest evidence for it.

Sam: So the thesis holds.

Alex: The thesis holds. But — and here's the uncomfortable part for the house — that edge is built on things that are, by their nature, copyable, and being copied. Harness patterns that leaked. A standard that was given away. A product lead in a market where a well-resourced rival ships a credible open alternative.

Sam: Whereas the one thing a competitor can't just read off a source map —

Alex: — is a genuinely better model. And that's the one place the gap right now is smallest.

Sam: So the uncomfortable question for the house is whether you want to be betting the company on an advantage in the most imitable layer there is.

Alex: And the honest answer is that it's a strong lead with a shorter half-life than its most enthusiastic backers admit. It's a real moat. But it's dug in sand that the tide keeps reaching.

Sam: Okay. Let's bring it home, because we covered a lot and I want the two or three things I'd actually walk away with.

Alex: Let's do it. Takeaway one: for two years we told a story about models and mostly meant a story about harnesses. The same model can swing from forty-six to eighty percent purely on the loop wrapped around it. The frontier weights have converged to within a point of each other. And the layer actually deciding whether your AI works is the one most people can't even name.

Sam: Takeaway two: that layer is now, for the first time, fully visible — the Claude Code leak saw to that — and partly imitable. Which is why DeepSeek could ship a credible open rival in August. And the honest grade on "it's based on the leaked code" is: the leak is real, the derivation is not.

Alex: Keep those two apart. That distinction is the difference between reporting and rumour. Takeaway three: the harness is splitting in two. The reasoning half has peaked and is receding into ever-stronger models that plan in their own heads. The product-and-environment half has not, and won't soon.

Sam: And the best current case for Anthropic is exactly that split — a better harness and product, co-designed with a model that's merely tied at the top rather than ahead. A real lead, in the most copyable place to have one.

Alex: Watch the harness. It was always half the story. It just spent two years hiding behind the model.

Sam: And honestly, that's the thing I'll carry out of this — how much of what we call "progress" is happening in a layer nobody even put a name to. I hope you're coming away seeing a bit more clearly where all of this is heading. It's a fast, genuinely complex picture with a brutally short shelf life on what you think you know — which is exactly what makes it worth following closely.

Alex: And one honest note on how this show is made: it's AI-generated. AI moves too fast to keep up with, so Dan built a custom stack of AI tools to research, analyse, verify and illustrate the questions worth understanding — mostly to learn them himself, and he shares what he finds. AI-assisted, fact-checked, and always worth a second look.

Sam: Before you go, one genuinely useful thing you can do: follow the show. Whatever app you're listening in right now, there's a follow or a plus button — one tap, and it's free.

Alex: And it does two things. You'll get each new episode the moment it lands, and for a small, independent show like this one, a follow is honestly the single biggest lever there is for helping it reach other people trying to make sense of all this.

Sam: So if this was worth your time, go ahead and hit follow. And one last thing on your way out — if there's someone in your life who keeps asking where AI is actually heading, send them this episode.

Alex: Genuinely one of the kindest things you can do — for them, and for the show. It's still small and independent, and every share does more than you'd think.

Sam: Thanks so much for listening. We'll see you in the next one.

Alex: See you then.