Inference at the Edge: Why Latency-Sensitive AI Belongs Closer to Your Users

For twenty-five years, the job of the edge was simple to explain: move your content closer to the people asking for it. Cache the image, the script, the video near the user, and the round trip to a distant origin stops mattering. It is the idea Akamai was built on, and the one every performance conversation still starts from.

The thing that sits far away has changed. For a growing slice of what your application does, the slow part is no longer a file waiting in a data center. It is a model, doing the thinking, in one centralized region that might be an ocean away from the person waiting on the answer. When your fraud check, your product recommendation, your support agent, or your search result is generated by an AI model, the physics that made a CDN worth having apply all over again, except now they apply to inference, not just content.

We have made the case before that latency is revenue, not a vanity metric. This is the same argument, one layer deeper. The question in 2026 is no longer only “where does your content live?” It is “where does your model run, and how far does every token have to travel before your customer sees it?”

The short version: inference, which means running a trained model rather than training it, is now roughly two-thirds of all AI compute, up from about a third in 2023, on Deloitte’s 2026 numbers. Unlike training, inference is user-facing and latency-bound: every token is generated in sequence, and every round trip to a distant region is time your user feels. For real-time work (agents that make many calls, personalization, fraud scoring at login, RAG, vision), that distance is a tax you are paying on every request. The fix is the same one that worked for content: run the latency-sensitive part closer to the user, and keep the heavy, batch-friendly part in the big central clusters where it is cheapest. Akamai’s Inference Cloud, and the NVIDIA AI Grid orchestration it launched in March 2026, route inference across 4,400+ edge locations, regional cloud GPUs, and dedicated Blackwell clusters based on cost and latency. Honest caveat up front, from the same Deloitte research: most inference still runs in central data centers, and it should. The edge is not where you run everything. It is where you run the part that can’t afford the trip.

Inference latency is the new performance budget

The shape of AI workloads has quietly flipped, and it matters for where the compute belongs.

Training a model is a bounded, batch job. It happens once, or periodically, in a big cluster, and no customer is sitting on the other end of it waiting. Inference is the opposite in every way that counts: it runs continuously, it is triggered by a user or an agent acting for one, and someone is almost always waiting for the result. Deloitte’s 2026 prediction puts inference at roughly two-thirds of AI compute this year, up from about half in 2025 and a third in 2023. The center of gravity has moved from the training run nobody sees to the live request everybody feels.

That live request has an unusual latency profile, and it arrives in two phases that bottleneck on different things. The first, prefill, reads your whole prompt and produces the first token; it is compute-heavy, runs in parallel, and sets your time to first token, the pause before anything appears. The second, decode, generates the rest of the answer one token at a time, and every token has to read the full model weights and the growing key-value cache back out of memory, so it is limited by memory bandwidth more than by raw compute. That split is why a model can start fast and then stream slowly, or the reverse, and why “buy a bigger GPU” is rarely the whole answer.

Distance sits on top of both phases. Before either one runs, the request has to travel to wherever the model lives and the answer has to travel back, and between distant cloud regions that round trip alone is often 70 to 150-plus milliseconds, going by Azure’s published inter-region latency, before a single token is computed. A multi-step agent that makes ten or twenty calls to finish one task pays that toll on every hop.

For a single question with a patient user, none of this is fatal. The problem shows up exactly where the money is: the interaction that has to feel immediate. We have watched teams spend months shaving Core Web Vitals down to the millisecond on the front end, then bolt on an AI feature whose answer is generated three thousand miles away and wonder why the page feels slow again. The slow thing stopped being the page. It became the model, and the model is often the furthest thing from your user in the whole stack.

Centralized versus distributed: where the milliseconds go

None of this is an argument that inference should all move to the edge. It shouldn’t, and the honest version of this pitch says so plainly.

Big centralized GPU clusters exist for good reasons. They give you the best cost per token on heavy, sustained work, they are where frontier models are trained and post-trained, and they can pack density that a small edge location never will. The same Deloitte research that flags inference overtaking training is equally clear that most inference will keep running in central data centers on expensive, power-hungry chips, not on cheap hardware at the edge. Anyone telling you the data center is finished is selling something.

The distributed case is narrower and more precise. It is about the specific workloads where the trip itself is the bottleneck: where the model isn’t especially large, the answer has to arrive in the moment, and the volume is high enough that the round trip, repeated across millions of requests, becomes the dominant cost in the experience. Akamai’s own framing when it launched the AI Grid in March 2026 draws the line about where I would: “AI factories have been purpose-built for training and frontier model workloads, and centralized infrastructure will continue to deliver the best tokenomics for those use cases,” said Adam Karon, its cloud chief, “but real-time video, physical AI, and highly concurrent personalized experiences demand inference at the point of contact, not a round trip to a centralized cluster.”

What is the trip actually worth? Akamai says its cloud inference delivers up to 3x better throughput, up to 2.5x lower latency, and up to 86% lower cost than traditional hyperscaler infrastructure for the workloads it targets. Treat those as directional rather than gospel. They are the vendor’s own figures, from its March 2025 cloud inference launch, not an independent benchmark, and your mileage will vary with model size and traffic pattern. But the direction is not controversial and the mechanism is plain: cut the distance and the queue for a premium GPU you didn’t need, and both the clock and the bill come down. The engineering question is which of your workloads actually live in that band, and that is a measurement, not a slogan.

The three tiers: far edge, regional GPU, dedicated clusters

The useful mental model is not “edge versus cloud.” It is a continuum, and the skill is placing each workload on it. Akamai’s platform happens to map cleanly onto three tiers, which makes it a good illustration of the general pattern.

The interesting part is not the tiers themselves but what moves work between them. Akamai’s Inference Cloud is, in its own words, the first global-scale implementation of NVIDIA AI Grid: an orchestration layer that routes each request to the right tier in real time, weighing cost, latency, and performance, using techniques like semantic caching (match each new prompt against ones you have already answered by embedding similarity, and serve the stored response when they are close enough instead of paying to generate it again) and intelligent routing (reserve the expensive GPU cycles for the requests that genuinely need them). At the far edge, that runs alongside the serverless compute (the EdgeWorkers and WebAssembly functions that teams already use to shape traffic before it hits an origin).

For an Eknix client, this is less exotic than it sounds, because the foundation is already under your feet. Our cloud infrastructure practice runs on Akamai Connected Cloud, the same platform the GPUs and the orchestration sit on. Adding inference to it is an extension of the edge you already operate, not a second platform to stand up and babysit.

What actually belongs at the edge

The band that benefits from distributed inference is real but specific. Five kinds of workload keep landing in it.

Agents that make many calls. An autonomous agent completing a task rarely makes one model call; it makes a chain of them, each depending on the last. Latency doesn’t add, it compounds. Twenty sequential calls with a fat round trip on each turns a task that should feel instant into one the user watches spin. As the agentic traffic on commerce sites climbs, the per-hop latency of the model behind them becomes a direct constraint on how much an agent can do before a human gives up.

Fraud scoring and personalization at login. Akamai describes financial-services customers using the grid for “rapid recommendations in the critical moments when customers log in,” which are decisions that have to resolve in the gap between a login and the first screen rendering. Score the session too slowly and you either hold up a legitimate customer or wave through a risky one. This is the same instinct behind scoring logins at the edge that runs through our account-takeover work; inference is just another signal that needs to arrive before the page does.

Retrieval-augmented generation. RAG pipelines fetch context, then generate against it. When the retrieval and the model both sit near the user, the whole loop tightens; when either is a continent away, the loop drags. For customer-facing RAG (support, search, product Q&A), regional placement is often the difference between a feature people use and one they abandon.

Real-time media and vision. Gaming studios are already deploying sub-50-millisecond inference for AI-driven characters, and broadcasters are transcoding and dubbing on the distributed network in real time. Anything where a human perceives lag directly, like an in-game character that stutters or a live translation that lands a beat late, has no tolerance for a distant round trip.

In-store and physical AI. Retailers are putting inference near the point of sale for associate tools and in-store applications, where the compute needs to be close to a physical place rather than a central cloud. The pattern generalizes to any deployment tied to a location instead of a data center.

Notice what is not on the list: training, frontier-model serving, and anything genuinely batch. Those belong in the core, and pretending otherwise wastes money. The discipline is telling the two groups apart honestly, which is the same vendor-neutral instinct we bring to every architecture conversation. The goal is the right tier for each workload, not the most edge you can bill for.

What to measure before you move anything

You cannot place a workload sensibly without numbers, and most teams have never captured the ones that matter for inference. Four are enough to start, and they are the same four Akamai’s orchestration optimizes for under the heading of “tokenomics.”

• Time to first token (TTFT). The pause before anything appears, and the one users read as “is it working?” It is set mostly by prefill and by how far the request had to travel. A few hundred milliseconds feels responsive; past about a second, attention starts to wander.
• Tokens per second. How fast the answer streams once it starts. A person reads around 240 words a minute, roughly four a second, and a token is about three-quarters of a word, so streaming much below five or six tokens a second feels slower than the reader can go.
• p95 latency, end to end. Not the average; the tail. The slow 5% of requests are where you lose people, and averages hide them. If you take one habit from our performance audits into your AI stack, make it this one.
• Cost per million tokens, by tier. The economic twin of latency. Run everything on premium central GPUs and the bill balloons; run everything at the edge and you overpay for capacity the workload didn’t need. The point of the continuum is matching each workload to the cheapest tier that still meets its latency target.

Baseline those four for each AI feature, per tier, and the placement decisions largely make themselves. The workloads with a tight TTFT requirement and heavy volume move toward the user. The batch-tolerant, cost-dominated ones stay central. The interesting middle, the ones that could go either way, is exactly where an architecture review earns its keep, because that is where the wrong default quietly costs you either money or milliseconds every day.

The reframe worth holding onto is that the edge is turning into an inference platform, not just a content one, and the logic that made a CDN obvious in 2005 is what makes distributed inference obvious in 2026. The heavy, batch, once-in-a-while work belongs in the big central clusters. The real-time, high-volume, user-facing work belongs close to the person waiting on it. Everything hard about this is in drawing that line correctly for your specific workloads, and in operating both sides of it once you have.