Apple Spends 15x Less on AI — and Read the Future Right

Transcript

Sam: Okay. The single most expensive thing humanity has ever built — and we're building it for a job that's busy becoming free.

Alex: That's the whole story in one sentence. This year the biggest companies on Earth will spend more than a trillion dollars building AI compute. The first trillion-dollar year of capital spending on one kind of infrastructure, ever. And the actual work that compute does is getting about ten times cheaper every single year.

Sam: So we're pouring concrete for the most expensive cathedral in history, and the congregation is quietly learning to pray at home.

Alex: Press play. Let's get into it. Welcome back to Dan's AI Intel — the show where we try to make sense of the fastest, most consequential shift any of us is likely to live through. I'm Alex, here with Sam, and today we're getting into something I think is the most under-told story in AI right now: where intelligence actually lives. Not which model is smartest — where the work physically happens. In a giant data centre somewhere else, or on the device already in your hand.

Sam: And that sounds almost academic until you realise the money riding on the answer. We're going to keep coming back to that opening tension all episode, because it really is the spine of the whole thing: a trillion dollars going one direction, and the actual workload sliding the other way.

Alex: The reason this matters — and the reason it's worth a whole episode — is that where the compute lives decides almost everything downstream. Who captures the money. What a product can cost. How private your data is. How fast an answer comes back. Which handful of companies hold real power as AI becomes ambient, just part of the air. A revolution that runs in five companies' data centres is a completely different revolution from one that runs on the billions of devices people already carry around.

Sam: So here's the map for today, and I want to lay it out because the shape of it is the argument. We start with the paradox — the trillion-dollar number, sitting right next to the fact that the work is collapsing in price. Then we go to why it's collapsing — there's a name for it, and it's a beautiful one. Then: can the small models actually do the job now, or is that hype? Then the people for whom this is already real — and it's not who you'd guess. Then the products that already shipped this year. Then the catch — because there's a brutal catch nobody puts in the tutorials. Then a genuinely useful decision rule for what to run where. And we finish on the company everyone says lost the AI race, which I'm told read the whole board more clearly than anyone.

Alex: And I'll just plant a flag on that last one, without giving it away: the company we end on is spending more than ten times less than its rivals on this, and being mocked for it. By the end you'll have a view on whether that's a blunder or the sharpest call in the industry.

Sam: A trillion dollars of "obviously" against one company's "obviously not." I know which way my gut leans, and I suspect by the end it'll have flipped. If you've been enjoying the show, follow us on Spotify or Apple Podcasts so you don't miss the next one — we do one of these properly, every week.

Alex: Let's start with the number, because honestly it's almost too large to feel. In 2026, total capital spending on compute across the whole industry — the big cloud providers, the specialist "neoclouds," national sovereign programmes, China, all of it — comes to roughly one-point-zero-four trillion dollars. By the Futurum Group's accounting, that's the first trillion-dollar year of compute capex in human history.

Sam: Give me a sense of who's actually writing those cheques, because "the industry" is doing a lot of work in that sentence.

Alex: Fair. Just the four biggest American hyperscalers between them are planning to spend on the order of seven hundred and twenty-five billion dollars. Amazon's guiding to around two hundred billion. Microsoft near a hundred and ninety. Alphabet's targeting a hundred and seventy-five to a hundred and eighty-five. Meta's in the hundred-and-fifteen to a hundred-and-thirty-five range. And of that whole pile, about three-quarters — call it four hundred and fifty billion dollars — points directly at AI infrastructure.

Sam: Four hundred and fifty billion. In one year. Aimed at one thing.

Alex: One of the largest concentrated deployments of capital any industry has ever attempted. And here's what every dollar of it is really saying: it's a bet that demand for centralised, cloud-based AI compute keeps climbing steeply enough to justify the build. That's the assumption baked into the concrete.

Sam: Okay, so that's bet number one. You said there's a second fact that sits awkwardly next to it.

Alex: From the same year. The price of running a model at any fixed level of capability is falling by about ten times every year. That's Andreessen Horowitz's figure. And by Epoch AI's measurement — and they look across a lot of different capability thresholds — the median is closer to fifty times a year, accelerating to two hundred times for some tasks since early 2024.

Sam: Wait. So the spending is exploding, and the cost of the work is collapsing — at the same time. Those feel like they can't both be true.

Alex: They're both true. And the way you reconcile them is one word: volume. The bull case says demand is growing so fast that even if the cost per unit drops a hundredfold, usage rises a thousandfold, so the absolute spend still climbs. And that might genuinely hold.

Sam: But.

Alex: But it papers over something structural hiding inside the aggregate. Because not all of that exploding demand needs to be served from a multi-billion-dollar data centre.

Sam: Right — "demand is up" and "all of that demand belongs in the cloud" are two completely different claims, and the trillion dollars only makes sense if the second one is true.

Alex: That's exactly the move. Here's the paradox stated plainly. The industry is pouring its trillion dollars into the cloud half of AI compute at precisely the moment a large and growing share of the actual workload is becoming cheap enough, and small enough, to run somewhere else entirely. On the edge. On your devices. Off the metered cloud.

Sam: The cathedrals are going up for a congregation that's learning to pray at home.

Alex: That's the line, and I want to be careful with it, because it's not "the cathedrals were a mistake." The hardest workloads, and training the frontier models in the first place, genuinely need them. The point is narrower and sharper: the assumption that all AI compute consolidates upward, into the cloud, is wrong. And the people building only for that future are exposed to a second future arriving underneath them.

Sam: So to see whether that second future is real, we have to go look at how far the floor has actually fallen. Which has a name, you said.

Alex: It has a wonderful name. The single most important economic fact in AI is not how good the best model is. It's how fast the cost of "good enough" is collapsing. And Andreessen Horowitz gave it a name — LLMflation. The inverse of inflation: the same dollar buys you dramatically more intelligence every year.

Sam: I love that, because inflation is the thing we all feel as our money getting weaker. This is your money getting stronger, but only when it's spent on intelligence.

Alex: And the headline number is stark. GPT-3, when it launched back in November 2021, cost about sixty dollars per million tokens. The cheapest model of broadly equivalent capability today — a small open model, something like Llama 3.2 3B, served through a commodity provider — runs around six cents per million tokens.

Sam: Sixty dollars to six cents. Let me just sit with that. That's a thousand times cheaper.

Alex: A thousandfold drop in three years. Which annualises to roughly ten times a year, for a constant level of capability.

Sam: And that's the part I want to nail, because it's easy to wave past. We're not saying the models got ten times better. We're saying for the same capability — the thing GPT-3 could do — the price fell by ten times a year.

Alex: Exactly. Capability held fixed, price in freefall. That's the measurement.

Sam: You mentioned Epoch had a more careful version of this.

Alex: They did, and the nuance is the good part. Epoch measured across many capability milestones and found the decline is real but uneven — their phrase is "unequally across tasks." It ranges from about nine times a year at the slow end, all the way to nine hundred times a year at the fast end. Median around fifty times a year, which jumped to two hundred times for some benchmarks after January 2024.

Sam: Nine times to nine hundred times. That's a massive spread. What decides where a given task lands in that range?

Alex: This is the bit that matters most. The cheapest, most commoditisable tasks fall fastest. The genuinely frontier capabilities hold their price the longest. So for instance, the price to reach GPT-4's level on hard, PhD-grade science questions fell about forty times a year — fast, but not nine-hundred-times fast.

Sam: So the floor is dropping out from under the easy stuff, and the ceiling — the really hard reasoning — is staying expensive.

Alex: And that, right there, is the engine of the whole two-tier world we keep gesturing at. The bottom collapsing toward free while the top stays costly. It's not one market anymore. It's splitting into two.

Sam: Can I just test that, though? Why would the easy stuff fall fastest? Intuitively I'd expect everything to get cheaper at roughly the same rate.

Alex: It's a great question, and the answer is competition. The cheap, commoditisable tasks — classification, extraction, summarising — lots of small open models can do those. So the moment one provider drops the price, three others undercut them, and the floor just caves in. The genuinely hard frontier reasoning? Almost nobody can do it. There's no competitive pressure dragging that price down, so it holds.

Sam: So scarcity protects the top, and a price war demolishes the bottom. The cheap stuff falls fast precisely because it's not special.

Alex: That's the whole texture of it. And it means the gap between "free intelligence" and "expensive intelligence" isn't closing — it's widening into two genuinely different markets, with different physics.

Sam: A curve that steep doesn't just happen. What's actually driving it?

Alex: Andreessen Horowitz pulls out six forces, and the important thing is they stack on top of each other. One: better price-performance from each new generation of GPU. Two: aggressive quantisation — and I'll explain that one. Three: software and serving optimisations. Four: smaller models reaching capability that used to need big ones. Five: better instruction-tuning, getting more out of each parameter. Six: brutal open-source competition compressing everyone's margins.

Sam: Hold on — quantisation. Translate.

Alex: So a model is, under the hood, a giant pile of numbers — the weights. Normally each of those numbers is stored at sixteen bits of precision. Quantisation is squeezing them down toward four bits — much coarser numbers — with surprisingly little loss in quality.

Sam: So it's like... taking a huge, ultra-high-resolution photo and saving it as a smaller file. Same picture, a fraction of the size, and your eye can barely tell.

Alex: That's a genuinely good way to hold it. Same model, far less memory and compute to run it, quality basically intact. And here's why I flagged it. None of those six forces is a one-off — they compound, year on year. But several of them — quantisation, smaller models, open weights — are exactly the forces that make a model small and cheap enough to leave the data centre in the first place.

Sam: Oh, that's the sleight of hand. LLMflation isn't just lowering the price of cloud tokens.

Alex: It's manufacturing the very models that no longer need the cloud at all. The same process that makes intelligence cheaper is also making it portable. And the clearest place to actually see that is in what a genuinely small model can now do — which two years ago, the answer was: not much. Two years ago, "small model" basically meant "toy." A few-billion-parameter model could autocomplete a sentence and not much else. Anything that looked like reasoning, or reliable tool use, or multi-step work, demanded a big frontier model reached over an API.

Sam: And now?

Alex: That gap has closed faster than almost anyone expected. And the reason is as much architectural as it is just scale. The breakthrough is something called sparsity — mixture-of-experts designs.

Sam: Okay, "mixture of experts." That's one of those phrases that sounds like it means something and I'm not sure it does. What is it actually doing?

Alex: It decouples how much a model knows from how much computation any single word costs it. Take Qwen2.5-Coder, a popular open coding model. It carries thirty-point-five billion total parameters — that's its total knowledge. But it only activates about three-point-three billion of them per token.

Sam: So it knows like a thirty-billion model but runs like a three-billion one.

Alex: That's it exactly. It runs at roughly the speed of a three-billion-parameter model while reasoning closer to a thirty-billion one — with a context window big enough to hold a real codebase. And the 2026 successors push the trick further: a thirty-five-billion-parameter model that fires only about three-and-a-half billion at a time.

Sam: Give me the analogy, because I think there's a clean one here.

Alex: Think of a huge hospital. It might have a thousand specialists on staff — that's the total knowledge. But you, walking in with one specific problem, don't get seen by all thousand. You get routed to the two or three who handle exactly your case. The hospital is enormous; your visit is small and fast. Mixture-of-experts is that: a giant model, but each question only ever wakes up the handful of "experts" inside it that it actually needs.

Sam: That's lovely, because it explains how you get both at once — the breadth of a big model and the speed of a small one. It's not a compromise between them.

Alex: And the dense small models matured too — the ones that aren't using that trick, that fire all their parameters every time. Microsoft's Phi-4, at fourteen billion parameters, scores about eighty-four-point-eight percent on the MMLU knowledge benchmark and fits comfortably on a consumer twelve-gigabyte graphics card. And the three-to-four-billion class — Phi-4-mini, Gemma 3 4B, Qwen3 4B — now handles real production coding help and document work on a laptop most developers already own. Phi-4-mini generates fifteen to twenty tokens a second on a MacBook Air.

Sam: A MacBook Air. Not a server rack. The thing in someone's bag.

Alex: But here's the capability that actually changed everything, and it's not raw knowledge. It's agency.

Sam: Meaning the model can do things, not just know things.

Alex: Right. The single biggest shift of 2026, the way the open-model community itself frames it, is that open and small models now reliably support the agentic patterns that until recently were the exclusive preserve of the big proprietary APIs. Function calling. Structured tool use. The plumbing — the model-context-protocol stuff — that lets a model actually act on the world rather than just talk about it.

Sam: How do you even measure "can it act," though? That sounds soft.

Alex: There's a benchmark built specifically for it — τ²-bench — and it tests exactly that: can the model call tools, execute steps, and recover when something goes wrong, across a multi-step workflow. Open coding models now score in the mid-eighties on it.

Sam: And that's the line between a clever autocomplete and an actual worker.

Alex: That's the threshold. And the part of that benchmark that matters most is the last bit — "recover from errors." Because calling a tool once when everything goes right is easy. The thing that separates a worker from a toy is what happens when the tool call fails, or returns garbage, and the model has to notice, back up, and try something else. That's the difference between something you babysit and something you can actually leave alone.

Sam: So it's not "can it do the step." It's "can it survive the step going wrong."

Alex: And once a small model can do that reliably, the economics get almost silly. Because remember — it's ten to thirty times cheaper to serve than a frontier model. But there's a second number that's just as important: you can fine-tune one of these in hours, not weeks. So you can take a small open model, train it on your specific narrow task overnight, and have a specialist that's cheap, fast, and better at your exact job than the giant generalist ever was.

Sam: Overnight. So this isn't some research-lab luxury. A normal team could actually do that.

Alex: A normal team with one decent graphics card could do that. Which is exactly why this stopped being theoretical. And NVIDIA's research arm put the whole conclusion into a title that reads like a thesis statement: "Small Language Models Are the Future of Agentic AI."

Sam: NVIDIA said that? The company whose entire business is selling the giant chips for the giant data centres?

Alex: Hold onto that, because it gets even sharper later. The paper — Peter Belcak and colleagues — argues small models aren't a budget compromise for agents. They're the correct tool, on three grounds at once. They're powerful enough for the narrow, repetitive tasks agents actually do. They're inherently more suitable, because those tasks have low variation. And they're necessarily more economical — ten to thirty times cheaper to serve than a frontier model, and fine-tunable in hours instead of weeks.

Sam: And the key word in there is "repetitive." The whole case rests on what agentic work actually looks like.

Alex: It's the hinge of the entire argument. So let's go look at it — at the people for whom this is already real. If you want to know where the local-compute future arrives first, here's the rule: watch the people building with AI, not the people chatting with it.

Sam: Why them specifically?

Alex: Because agentic workloads have an economic structure that makes local irresistible — and they have it right now, not in some projected future. Picture a chatbot. You ask it something, it makes one expensive call, it returns a paragraph a human reads. That's the chat shape.

Sam: One question, one answer.

Alex: An agent does something completely different. To complete a single complex task, it makes thousands of small calls. Most of them narrow and repetitive — parse this command, extract these fields, classify this code change, format this output, decide which tool to call next, summarise what just happened.

Sam: So the agent is having this frantic little internal monologue, thousands of tiny thoughts, just to do one thing I asked.

Alex: And NVIDIA's researchers describe it exactly that way — agentic systems are, in their words, "a mass of applications in which language models perform a small number of specialised tasks repetitively and with little variation." Each of those calls, on its own, is easy. None of them needs a frontier model.

Sam: But there are thousands of them.

Alex: There are thousands of them per task. So if you route every single one to a premium cloud API, the cost is enormous. And the flip side — not doing that — is the single largest lever in the whole economics of agents. Even a partial shift of those calls from a big model to a small one cuts operating costs by orders of magnitude, while leaving the result essentially unchanged.

Sam: Orders of magnitude. For basically the same output. That's not an optimisation, that's a different business.

Alex: And this is the same economic wound, seen from the other side, that makes selling frontier inference so brutal. We've talked before on the show about what an AI subscription really costs the company making it — how unforgiving the unit economics of premium reasoning models are. A heavy user, running long chains of thought all day, can burn far more compute than their subscription ever covers.

Sam: Right, the all-you-can-eat buffet where a few customers eat the whole kitchen.

Alex: Now invert it. Push that workload onto a small local model and the whole equation flips. The marginal cost of a token drops to basically the electricity it takes to compute it. The per-token meter just... disappears. And the code never leaves the laptop.

Sam: So for a builder running an agent that's hammering a model all day long, "free, private, on my own hardware" isn't a nice-to-have.

Alex: It's a structural cost advantage. It's a different cost base than your competitor who's renting every token. And the cultural shift is already visible. Andrej Karpathy — who has a sharper feel for where this is going than almost anyone — coined a term in early 2026 for his own working life: agentic engineering. His description of it: "You are not writing the code directly ninety-nine percent of the time. You are orchestrating agents who do, and acting as oversight."

Sam: So the human's job moved up a level. You're not the bricklayer, you're the foreman watching a crew of agents.

Alex: And step back and notice what that does to the whole race, because this is the deeper implication. For three years the industry has measured itself by one number — whose model sits at the top of the benchmarks. But if most of the actual work is commoditising and draining to the edge, then the frontier becomes a smaller and smaller slice of where the value actually lands.

Sam: So being king of the benchmarks matters less if the benchmark-grade work is the rare part, and the bulk has moved to cheap models on people's own machines.

Alex: And the durable advantage shifts. It stops being "who owns the best model" and becomes "who owns the distribution of cheap intelligence to where people already are." Which — keep that sentence in your pocket — is exactly the bet we end the episode on. But you can already feel it forming here, in how builders work: the prize is moving from the model to the place the model runs.

Sam: Okay, I'm officially holding that thought.

Alex: And his own experiments point the same direction. He released an autonomous research agent that runs machine-learning experiments unattended, on a single GPU — looping through propose, train, evaluate, seven hundred times over two days, no human in the loop. And a personal knowledge system he built on plain text files that he says is seventy times more efficient than the vector-database approach it replaced.

Sam: Seventy times — by going simpler. By using less machinery, not more.

Alex: And that's the pattern underneath all of it. Take the expensive, general-purpose frontier model out of the inner loop. Put cheap, specialised, often-local compute in the hot path — where the volume actually is. Builders aren't waiting for anyone's permission here. The tooling showed up to meet them. Which is the next question: what tooling? Because this stops being a prediction the moment you can download it. So here's why this is a 2026 story and not a 2028 prediction. The combined local-plus-cloud stack stopped being an architecture diagram and started being a product you can actually download. The design pattern even has a name now — hybrid inference. And the shape is clean: a small model running locally acts as a router. For each request, in real time, it decides — can I answer this on the device, or does this one need to go up to a frontier model in the cloud?

Sam: So the little local model is basically the receptionist. It takes every request, and most of the time it just handles it, and only the genuinely hard cases get sent through to the specialist.

Alex: That's the architecture, and it's exactly right. And the most striking demonstration came from Perplexity. At Computex 2026 they unveiled a hybrid local-server inference orchestrator — built with Intel, rolling out to their desktop app in July.

Sam: What makes it different from just "sometimes use a small model"?

Alex: The framing of the routing decision itself. The local model treats where to run each subtask as an inference problem — something to reason about in real time, mid-task, invisibly to you. Sensitive data — your financial records, health files — stays on the device by default. Genuinely hard reasoning goes to the cloud. And the system asks permission before anything sensitive ever leaves the machine.

Sam: So privacy isn't a setting you toggle. It's the default behaviour of the router.

Alex: And Perplexity's chief executive, Aravind Srinivas, was unusually blunt about why. He said he does not want "all your compute centralised in servers." He pointed out that "some people are spending half a billion dollars per month" on inference — and that pushing work onto users' own devices is a way to cut that bill.

Sam: Half a billion a month. That's the problem they're trying to make disappear.

Alex: And his company's own numbers make the case. Revenue grew from a hundred million dollars to five hundred million while headcount rose just thirty-four percent. And the blog post announcing the whole thing carried a title that honestly doubles as the thesis of this entire episode: "The data centre moves to your machine."

Sam: That's the cathedral line again, isn't it. Said by the people building it.

Alex: And I want to pull on that routing idea once more, because it's the genuinely clever part. The old way you'd build this is a dumb rule — "anything under N words, do it locally; otherwise, cloud." Brittle, and usually wrong. What Perplexity is doing is treating the question "where should this run" as itself a thing the model reasons about, fresh, for every single request, in the middle of the task.

Sam: So the decision isn't hard-coded. The system is genuinely thinking, per request, "can I handle this one, or is this one over my head?"

Alex: And it does it invisibly — you never see the handoff. And the privacy piece falls out of that for free, which is the part I think people will end up caring about most. Your financial records, your health files — those don't go anywhere by default. They're handled on the machine. The only time anything sensitive leaves is when the system stops and asks you first.

Sam: Which is the opposite of how the cloud era worked, where the default was "everything goes to the server and you hope they're careful with it."

Alex: Complete inversion. The default flips from "send it all up" to "keep it all here unless there's a reason not to." And that's not a privacy feature bolted on the side. It's just what the architecture does when the smart thing lives on the device.

Sam: But Perplexity's a challenger. They've got every reason to want to route around the giants. Is anyone with skin in the cloud game doing this?

Alex: That's exactly the right question, and the answer is the tell. Perplexity is just the most legible example of an industry-wide move. Microsoft shipped Foundry Local — a runtime of roughly twenty megabytes that runs models on a Windows machine's processor, graphics card, or neural chip, with no API key, no per-token cost, no cloud dependency at all. And at their Build conference they introduced Aion 1.0 Plan — a fourteen-billion-parameter model with a thirty-two-thousand-token context window and fully local agentic tool-calling and multi-step planning.

Sam: Microsoft. Who sells Azure. Who makes money when you run things in their cloud.

Alex: And NVIDIA pushes the same direction with its desktop AI hardware. And Google has wired a small Gemini model directly into Chrome. So sit with the shape of that. The largest companies in computing are, simultaneously and independently, building the on-ramp for AI work to leave their own data centres.

Sam: That's the part that actually convinces me. Because a challenger talking their book, fine. But when the companies whose revenue literally depends on cloud inference are themselves shipping the tools to move that inference off the cloud — they're not doing that for fun.

Alex: They're doing it because they can see where it's going, and they would rather sell you the shovel than miss the dig. When the incumbents build the off-ramp, the trend has stopped being speculative. There's only one thing standing between this stack and being everywhere. And it's a big one. The moment you go local — it gets genuinely, painfully hard to make it work. Here's the counterintuitive truth all the cheerful "run it on your laptop for free" tutorials tend to skip. The model was never the hard part of going local. The harness is.

Sam: Define "harness," because I don't think most people have a picture for that word.

Alex: The harness is everything around the model. All the surrounding software that takes this clever-but-unreliable component and makes it behave like dependable software you can actually ship. And here's the evidence for why it's the real problem. Across real-world deployments, roughly sixty-five percent of enterprise AI agent failures trace not to the model's reasoning at all — but to "harness defects."

Sam: Sixty-five percent. So most of the time the agent breaks, it's not because the model was dumb.

Alex: It's not the brain. It's the wiring. Three failure modes specifically: context drift, schema misalignment, state degradation. In plain terms — the system around the model loses track of what the model knows, or it mangles the format of a tool call, or it corrupts the agent's memory of its own task.

Sam: So the model says the right thing, and the plumbing fumbles it on the way to actually doing it.

Alex: And the blunt conclusion the field has landed on is striking: in a lot of deployments, improving the system around the model gets you better results than upgrading to a bigger model.

Sam: That's almost heretical, right? The whole industry's reflex is "throw a smarter model at it."

Alex: And the reason it's true is fundamental. A language model's compliance with your instructions is probabilistic, not deterministic. It usually does what you ask. Not always.

Sam: Which is fine for a chatbot and a disaster for software.

Alex: Right — and the only way to make a probabilistic component behave like dependable software is to wrap it in deterministic constraints. Linters. Type checks. CI gates. A plan-execute-verify loop that catches the model's mistakes before they spread.

Sam: Give me the analogy. What is the harness, really?

Alex: Think of an unbelievably talented intern. Brilliant, fast, occasionally just... confidently wrong. You would never let that intern push straight to production. You put checks around them — a senior reviews the work, tests have to pass, there's a checklist before anything ships. The harness is that whole apparatus of checks. It's the machinery that converts an unreliable genius into a reliable worker.

Sam: And the genius is free now. It's the apparatus that's expensive.

Alex: And here's the cruel twist. That tax — building all that scaffolding — is steepest exactly where you'd hope it would be lightest. On your own machine.

Sam: Why there? I'd assume my laptop is the simple case.

Alex: Because in the cloud, an army of engineers already built and tuned that scaffolding for you, invisibly. You inherit it polished. Go local, and you get it raw and unassembled. Now — running a capable coding agent against a local model is genuinely possible today. Since early 2026, the open-source Ollama runtime speaks the same API as the major coding agents, so tools like Claude Code, Cline, OpenCode work with local models right out of the box.

Sam: So the front door works. What goes wrong?

Alex: The lived experience is a cautionary tale about how much the cloud was quietly doing for you. Get the local model's configuration subtly wrong — and the defaults are wrong — and the agent "loses track of file contents mid-edit, forgets earlier instructions, and produces fragmented changes." It silently breaks a refactor in a way that looks like success until you actually read the code.

Sam: Looks like success. That's the nightmare, isn't it — not the error you can see, the one you can't.

Alex: And the single most common failure is so mundane it's almost funny, and it's brutal. The runtime's default context window is just four thousand tokens.

Sam: Meaning the model can only hold about four thousand tokens of the conversation in its head at once.

Alex: And an agentic coding session blows through four thousand tokens in seconds. So the model just... forgets the first half of its own task. Unless you happen to know to go in and raise that setting to thirty-two thousand or more.

Sam: So there's this invisible cliff, and if nobody told you the setting exists, you walk straight off it and the thing quietly falls apart — and you blame the model.

Alex: And none of that is the model's fault. It's the harness — the part that, in a polished cloud product, a whole team already built and tuned for you, and that you inherit raw the second you go local.

Sam: So this is the real gatekeeper. Not "is the model good enough."

Alex: This is the gate on the whole edge future, and it's why the transition is uneven rather than sudden. The economics of local compute are overwhelming and getting stronger every quarter. But economics don't deploy themselves. Engineering does.

Sam: So who actually wins the local-AI era? Because it sounds like it's not whoever has the best small model.

Alex: It's not — those are converging and increasingly free. The winners will be whoever builds the best harness around a small model. The orchestration, the verification, the routing, the error recovery — all the boring, deterministic plumbing that makes a probabilistic component trustworthy in production.

Sam: And that's a software problem. Not a model problem.

Alex: It's a software problem, and it is wide open. Nobody's locked it down yet.

Sam: Let me make sure I've actually got why the harness beats the bigger model, though, because it still feels backwards. If my model is smarter, surely it makes fewer mistakes, so I need less scaffolding?

Alex: You'd think so. But here's the thing — a smarter model is still probabilistic. It's wrong less often, but it's still wrong sometimes, and "sometimes" is fatal for software. Whereas the scaffolding — the linter, the type check, the verify step — is deterministic. It catches the mistake every single time, no matter how rare. So a slightly-dumber model wrapped in a good harness beats a slightly-smarter model running naked, because the harness closes the gap the model can never fully close on its own.

Sam: So you're not trying to make the model perfect. You're building a net that catches it whenever it slips.

Alex: And the shape of that net has a name — a plan-execute-verify loop. The model proposes a plan. It executes a step. And then something deterministic checks the result before anything moves forward. Propose, do, verify, repeat. That loop is most of what separates a demo from a product. Which, if you're trying to decide where to place your own effort as a builder, is maybe the most useful sentence in this whole episode: the edge isn't the model, it's the net. So let's get concrete — what should you actually run where? Strip away all the noise, and the practical question facing any company or any engineer is concrete. Of all the AI work you do, how much should run on your own hardware versus a cloud API? And what should you actually pay for? And the good news is the answer has gotten usefully sharp. It has two axes. The first is task type.

Sam: Okay, so how do I look at a piece of work and know which bucket it's in?

Alex: The dividing line is reliability under variation. The work that belongs local is the high-volume, well-specified, low-variation work that needs competence but not brilliance. Classification. Extraction. Summarisation. Formatting. Routing. The routine tool-calls inside an agent. And — to take the cases a builder feels most — running test suites, triaging and labelling, routine repository chores, first-pass dependency and security scanning. The deterministic-with-a-little-reasoning tasks that recur thousands of times and basically never surprise you.

Sam: So the stuff that's high-volume and boring and predictable goes on your own hardware. What earns the trip to the cloud?

Alex: The minority that genuinely earns it. Dynamic planning. Complex multi-step reasoning. Novel problems. Sophisticated error recovery. The work where a frontier model's extra capability is the actual difference between success and failure.

Sam: And do we have a number for how that splits in practice? Because "most of it is routine" is the kind of thing people assert.

Alex: We do, and it's consistent and striking. Industry routing data says somewhere between seventy and eighty percent of production queries never need a frontier model at all.

Sam: Seventy to eighty percent. So the giant expensive model is genuinely overkill for the large majority of what's actually being asked of it.

Alex: And a well-tuned router — that sends that bulk to small, cheap, often-local models and reserves the frontier model for the genuinely hard reasoning — cuts total inference cost by sixty to eighty percent, with negligible loss in quality.

Sam: That number's wild. You're telling me there's a sixty-to-eighty-percent saving sitting on the table, and the only thing standing between a company and it is being willing to route the easy stuff to the cheap model.

Alex: That's the lever. That's the whole game in one decision.

Sam: You said two axes. What's the second?

Alex: The cost crossover. This is the one that answers "what do you actually pay for," in money terms. Cloud APIs are pure operating expense. You pay per token, forever, and you pay a margin on top of the provider's own costs.

Sam: And that word "forever" is doing some quiet damage there, isn't it. Because operating expense never ends. It's not that you bought the thing — you're renting it every single month, for as long as you run it, and the bill only goes up as you use it more.

Alex: And you're paying the provider's profit margin on top of their own costs, every token, the whole time. Whereas self-hosting is capital expense — a meaningful cost up front, but then amortised across everything you run on it. You buy the machine once; after that, the marginal cost is mostly just power.

Sam: Give me the real numbers, because "it depends" is doing a lot of hiding.

Alex: An eight-GPU server runs four hundred to five hundred thousand dollars. Or — and this is the part people miss — nothing beyond hardware you already own, for laptop-scale local models. And the crossover between those two worlds is a function of one thing: utilisation. How busy you keep the machine.

Sam: So it's like buying a car versus taking taxis. The car's a fortune up front, but if you drive constantly it's far cheaper per mile. Take two trips a year, and the taxi wins easily.

Alex: That is exactly the shape, and the numbers are concrete. Once a workload runs at sustained load above roughly sixty to seventy percent, self-hosted inference comes in eight to eighteen times cheaper per token than the equivalent cloud API. And the break-even against the cloud arrives in as little as a few months for a genuinely steady workload.

Sam: Eight to eighteen times cheaper. So for the heavy, constant stuff, owning the machine isn't a little cheaper. It's an order of magnitude.

Alex: And out of all of this falls a clean decision rule. Run it local when the load is high, steady, and predictable. When data residency matters. Or when the token volume is so large the per-token meter dominates everything. Reach for the cloud when the work is bursty or experimental. When you can't or shouldn't commit capital up front. When you need multi-region reach. Or when you need the very newest frontier capability the instant it ships, before any small model has caught up.

Sam: So the one-liner is: pay for the frontier, and pay for elasticity. And stop paying, by the token, for the commodity.

Alex: That's the whole thing distilled. Pay for the two things that are genuinely scarce — the frontier capability, and the ability to scale up and down on demand. And stop renting, by the token, the thing that's becoming a commodity.

Sam: And the lovely part is that's not a compromise you settle for. The hybrid isn't "well, we couldn't decide, so we did half and half."

Alex: No — and that's the reframe I want to land. It's the actual optimum. It is no accident that roughly seventy percent of enterprises are expected to run hybrid architectures in 2026. They're not hedging their bets. They've worked out that the split — frontier on top for the hard minority, cheap and local underneath for the volume — is simply the lowest-cost way to get the same quality. The hybrid isn't a hedge. It's the right answer.

Sam: Which sets up the company we've been circling all episode. Because if the model is becoming the commodity, somebody big must be betting their whole strategy on it.

Alex: One of the biggest. And they got mocked for it for two straight years. For two years, the consensus on Apple has been simple: they lost the AI race. No frontier model. No viral chatbot. An assistant that visibly lagged. And here's my claim — that consensus mistakes a game Apple chose not to play for a game it lost.

Sam: That's a big distinction. Unpack it, because from the outside "we don't have a frontier model" sure looks like losing.

Alex: Apple is making a deliberate, structural bet that the model is becoming the commodity and the edge is the moat. And in 2026 the shape of that bet became undeniable. Start with what they actually built — because their architecture is the whole two-tier world we've been describing, rendered in silicon.

Sam: Okay, walk me through it.

Alex: On the device, Apple's third-generation Foundation Models — detailed by their own researchers in June 2026 — are led by a model called AFM 3 Core Advanced. Twenty billion total parameters. But built sparse, so it activates just one to four billion of them at a time, depending on the request. It uses a pruning technique that keeps the full model sitting in flash storage and loads only the experts it needs into memory.

Sam: That's the hospital again. The mixture-of-experts trick from earlier — only it's shipped, in your pocket, running privately.

Alex: That's exactly what it is — that efficiency trick, productised, running privately on the phone. And above it sits a server tier, on Apple's Private Cloud Compute, for the hardest work. And Sam — here is the tell. The single most revealing fact in the whole story.

Sam: Go on.

Alex: Apple's most capable cloud model, AFM 3 Cloud Pro, is optimised for NVIDIA GPUs inside Google Cloud. And all five of Apple's foundation models were — in Apple's own words — "custom-built in collaboration with Google."

Sam: Wait, wait. Apple — the most valuable company on Earth, the company that controls everything down to the screws — asked to field a frontier model, went and built it with Google? A rival? And runs it on the rival's computers?

Alex: They put their own engineering into the on-device tier and the privacy architecture, and they leaned on Google for the cloud frontier. And just think about what that decision reveals. That is not the behaviour of a company that believes the biggest model is where the value is.

Sam: Because if you thought the frontier model was the prize, you would never, ever outsource it to a competitor. You'd die before you did that.

Alex: You'd spend anything to own it. Which brings us to the money — because the financials make the bet legible in a way nothing else does. Apple's 2026 capital expenditure is roughly thirteen billion dollars.

Sam: Okay, hold that next to the others for me again. Amazon was —

Alex: Amazon, roughly two hundred billion. Microsoft, roughly a hundred and ninety. Alphabet, a hundred and seventy-five to a hundred and eighty-five. And Apple — thirteen.

Sam: So Apple is spending more than ten times less than any single one of its hyperscaler rivals.

Alex: More than ten times less than any one of them. And around fifty times less than the four of them combined — on the AI infrastructure build that everyone else is treating as existential, as life-or-death.

Sam: A company that believed the future belonged to whoever owns the most cloud compute could not possibly spend that little.

Alex: That's the whole argument in one line. Apple spends this little because it does not believe that. The thirteen billion isn't Apple failing to keep up. It's Apple telling you, in the only language that can't lie — where it puts its money — that it thinks everyone else is building the wrong thing.

Sam: So if the moat isn't the model, what does Apple think the moat is?

Alex: Bank of America's Wamsi Mohan has the sharpest framing of it. He calls Apple's real asset an "agentic AI moat" — its control over the device, the user's identity, their payments, their trust. And — this is the key move — that moat becomes more critical, not less, "as AI agents proliferate" and the underlying model commoditises beneath them.

Sam: So Apple's logic is: when intelligence is everywhere and basically free, the model stops being the scarce thing.

Alex: And the scarce, defensible thing becomes the trusted surface it runs on. And Apple owns the most valuable such surface on Earth — a billion-plus devices it can upgrade with AI overnight, through a software update. In homes and pockets that no rival AI company can reach without first solving the hardware, the distribution, the ecosystem problems Apple already solved decades ago.

Sam: Everyone else has to fight their way to the device. Apple's already there. It's already in the hand.

Alex: And sit with the asymmetry of that for a second, because it's the part I find genuinely hard to argue with. For a rival AI company to get where Apple already is, they'd have to build a phone, build an operating system, earn a billion people's trust with their payments and their identity and their photos, and win the retail and the carrier deals — decades of work, the exact problems Apple ground through twenty years ago. For Apple to get an efficient model onto a billion devices, it ships a software update on a Tuesday.

Sam: That's the whole moat in one image. One side has a twenty-year head start it can't be sped past; the other side just has a really good model — and the model is the thing turning into a commodity.

Alex: And that's why the thirteen-billion number isn't timidity. It's a company that looked at where the value is actually going to pool, and decided to spend its money on the part nobody can copy — the trusted surface — instead of racing for the part that's becoming free.

Sam: You're making it sound airtight. It can't be airtight. What's the case against?

Alex: It's genuinely not guaranteed, and I want to be honest about that. Apple's on-device models are genuinely behind the frontier on hard reasoning. Its assistant has disappointed before — more than once. And leaning on Google for the cloud tier is a real dependency on a competitor that could bite.

Sam: So the bet could lose.

Alex: The bet could lose. But the strategic premise underneath it is, I think, the most clear-eyed read of this entire trend. If most AI interaction becomes ambient, private, and edge-resident — running quietly on the device — then the company that wins is not the one with the cleverest model sitting in a data centre. It's the one whose efficient-enough models are already running. Trusted, integrated, on the device in your hand.

Sam: So the line that sticks for me is: Apple isn't failing to play the AI race. It's betting the race everyone's watching is the wrong race.

Alex: That's it exactly. And you don't have to agree with them to take the lesson. You just have to notice that the company everyone wrote off looked at the same board as everyone else — and read it completely differently. And put thirteen billion dollars where its read was.

Sam: Okay. Let's bring it home. If someone's been walking and listening to all of this, what are the few things they actually carry out the door?

Alex: Three things. First — the trillion-dollar capex race is real, and the cloud frontier it's building is real and worth paying for. But it is not the whole map. And treating it as the whole map is the strategic error of the moment.

Sam: Because underneath that frontier, the floor is collapsing.

Alex: Second thing. The cost of competent intelligence is falling at roughly ten times a year. Small models have crossed the line into reliable agentic work. And the combined local-plus-cloud stack — a small model on your own hardware deciding what it can handle and what to escalate — has shipped, this year, from Perplexity, Microsoft, Apple, and Google. All in the same twelve months. That's not a forecast. It happened.

Sam: And the third?

Alex: The third is the operational one — the bit you can actually use on Monday. The move is no longer to send everything to a frontier API. It's to route the seventy to eighty percent of work that's repetitive and well-specified and high-volume to small, cheap, increasingly local models — and pay frontier prices only for the genuine reasoning that earns it. A split that cuts cost by sixty to eighty percent today, and tilts further toward the edge every quarter.

Sam: And the catch — so nobody walks away thinking it's free.

Alex: The catch is the harness. The thing that gates this future isn't the model, which is converging toward free. It's the deterministic engineering that makes a probabilistic component dependable — the orchestration, …