The Latency Tax: Why AI Infrastructure Geography Still Matters

← All Articles

The central claim of cloud computing was that geography would stop mattering. With fast enough networks and large enough facilities, compute could be concentrated wherever land and power were cheap, and users everywhere would experience something indistinguishable from local processing. For many workloads, that claim has been adequately true. For AI inference at scale, it is not.

The latency tax is the aggregate cost imposed on AI systems by the distance between where computation occurs and where its outputs are consumed or acted upon. It is not a single number. It compounds through every layer of the delivery stack: the speed of light, protocol overhead, queuing at intermediate nodes, and the mismatch between accelerator throughput and the rate at which results can be delivered to the systems that need them.

This essay traces the sources of the latency tax, examines why LEO satellite networks resolve some problems while leaving others intact, and describes the geography of infrastructure that actually minimizes the tax for AI workloads in high-demand coastal regions.

Where Compute Concentrates vs. Where Demand Lives

AI compute infrastructure has concentrated in a small number of locations: northern Virginia, the Pacific Northwest, and a handful of large European facilities account for a disproportionate share of deployed accelerator capacity. The factors driving concentration are well understood: large parcels of affordable land, abundant and cheap power, proximity to long-haul fiber routes, and favorable regulatory environments.

These factors are not correlated with where AI inference demand lives. Demand for low-latency inference, the kind generated by enterprise agents, real-time applications, physical automation, and interactive media, is concentrated in population centers. The largest population centers in the United States, Europe, and East Asia are predominantly coastal.

The mismatch between where compute concentrates and where demand lives is the fundamental geographic problem. A data center in central Oregon serves the Pacific Northwest adequately but imposes a 30 to 60 millisecond round-trip penalty on users in Los Angeles, San Francisco, or Seattle. A facility in northern Virginia serves the mid-Atlantic but penalizes users in Miami, Boston, or Atlanta. These penalties are small for asynchronous workloads. They are significant for real-time applications.

Population density along coastlines is not accidental. Coastal regions developed as centers of commerce, trade, and industry over centuries, and major metropolitan infrastructure, power grids, fiber networks, financial centers, hospitals, and ports are concentrated there. AI workloads that integrate with physical infrastructure inherit its geography.

Latency vs. the Latency Tax

Latency is the round-trip delay between a request and a response. The latency tax is a broader concept: it is the cumulative cost imposed by that delay on all of the systems and users that depend on the inference result.

For a single user asking a single question, a 60-millisecond latency increase is imperceptible. The latency tax on that interaction is trivial. For a real-time monitoring system that must process 1,000 events per second and take action on each within 100 milliseconds, a 60-millisecond network round trip consumes 60 percent of the entire processing budget before any computation begins. The tax is not trivial; it is structural.

The tax also compounds. When an AI pipeline involves multiple inference calls, each call incurs the full network round-trip cost. A pipeline with four sequential inference steps, a common pattern in agentic architectures, incurs four times the latency tax of a single step. At 60 milliseconds per round trip, a four-step pipeline has a floor of 240 milliseconds of network delay independent of how fast the inference itself completes.

A pipeline with four sequential inference steps incurs four times the latency tax. At 60ms per round trip, four steps impose 240ms of network delay before any inference begins.

Reducing the network round trip from 60 milliseconds to 10 milliseconds does not reduce the latency tax by 50 milliseconds. It reduces it by 200 milliseconds across the four-step pipeline. The benefit of geographic proximity compounds in proportion to the complexity of the AI pipeline.

Low-Earth Orbit Satellite Networks

LEO satellite constellations have changed the calculus for network access in underserved areas. A terminal connected to a LEO constellation can achieve latencies of 20 to 40 milliseconds to a ground station, compared to 500 to 600 milliseconds for geostationary satellite links. For users in areas without terrestrial broadband, LEO connectivity is a meaningful improvement.

The AI infrastructure discussion has imported the premise that LEO latency is competitive with terrestrial fiber for inference delivery. This deserves scrutiny. A LEO link to a user does not resolve the latency problem if the compute is still located in a distant terrestrial data center. The satellite delivers the user's request to a ground station, which then routes it over terrestrial fiber to the data center. The total path latency includes both the satellite segment and the terrestrial segment.

If the ground station is co-located with or adjacent to the compute facility, and the user is within LEO coverage of that ground station, the total round-trip can be competitive with a direct terrestrial fiber connection from a similarly distant user. But this requires the compute and the ground station to be in the same geographic region as the user population being served. The satellite resolves the last-mile terrestrial connectivity problem; it does not relocate the compute.

Surface Networks and the Satellite Path

The path from a user to an AI inference endpoint typically involves multiple network segments: the user's local network, an access network, one or more peering points, and the network connecting the last peering point to the data center. Each segment adds latency and introduces variability.

LEO satellite paths add one additional segment: the uplink from the user terminal to the satellite, the inter-satellite link if the packet traverses the constellation, and the downlink to a ground station. Each segment adds latency and introduces variability in ways that differ from terrestrial fiber.

For AI inference, variability is as important as average latency. A system designed for 100-millisecond response time budgets can tolerate average latency of 40 milliseconds but may fail if 5 percent of requests exceed 150 milliseconds due to satellite path variability. The tail latency of LEO satellite links under load is an important parameter for AI application design, and published performance data from commercial operators, including Starlink, typically characterizes median and 90th percentile latency rather than 99th percentile performance under sustained load.

The Orbital Latency Floor

Physics establishes a lower bound on LEO satellite round-trip latency. A satellite in a 550-kilometer orbit is approximately 550 kilometers above the Earth's surface at zenith. The round trip from terminal to satellite and back is at minimum 3.67 milliseconds for a signal traveling at the speed of light. In practice, signal propagation in the atmosphere, terminal processing, and satellite routing overhead bring the realistic minimum round trip to 20 to 30 milliseconds even under ideal conditions.

For a user that would otherwise reach a coastal data center in 8 to 12 milliseconds via fiber, the LEO satellite path adds latency rather than removing it. The orbital floor is not a hypothetical concern: it represents a real constraint on the use cases where satellite delivery of AI inference is competitive with well-placed terrestrial infrastructure.

The use cases where LEO satellite delivery is genuinely advantageous are those where terrestrial alternatives are unavailable or prohibitively expensive: maritime operations, remote industrial sites, rural coverage areas, and aviation. For dense urban and suburban users within reach of coastal infrastructure, terrestrial connectivity to geographically proximate compute is lower latency than satellite delivery to distant compute.

Where Long-Distance Satellite Has an Edge

Inter-satellite links in LEO constellations propagate signals in vacuum rather than through fiber. The refractive index of optical fiber is approximately 1.5, meaning signals travel through fiber at roughly two-thirds the speed of light in vacuum. Signals propagating through vacuum via inter-satellite links travel at the full speed of light.

This difference creates a window where LEO inter-satellite routing is faster than the equivalent terrestrial fiber path for sufficiently long distances. Estimates place the crossover distance at roughly 3,000 kilometers: for paths longer than this, well-routed LEO constellation paths can have lower latency than fiber.

The practical significance for AI infrastructure is limited but real. Transatlantic or transpacific paths for AI inference that genuinely cannot be served by regional compute near the user are candidates for LEO routing. But the correct first-order solution is to place compute near the user, reducing or eliminating the long-distance path entirely. LEO routing is a performance improvement on a long-distance path; eliminating the long-distance path is a structural improvement.

LEO routing is a performance improvement on a long-distance path. Eliminating the long-distance path is a structural improvement.

The Throughput Tax

Latency is the most frequently discussed geographic cost, but it is not the only one. Throughput, the rate at which data can be delivered between two points, is also constrained by distance and network topology.

AI workloads that involve large input or output payloads, video inference, multimodal models receiving high-resolution imagery, or systems returning large embeddings or generated content, consume significant bandwidth on the path between user and compute. Long network paths with multiple intermediate nodes are subject to greater congestion variability, lower effective throughput under load, and higher sensitivity to packet loss than short paths with fewer intermediate hops.

The throughput tax manifests most clearly in high-volume inference scenarios: a system processing thousands of video frames per second, or a large number of concurrent users each sending multi-megabyte inputs. In these scenarios, the network becomes a bottleneck independent of how much accelerator capacity is available at the destination.

Bandwidth Cost and Data Egress

Data transfer out of major cloud regions is priced at rates that can add significantly to the total cost of AI inference at scale. Egress pricing reflects the real cost of operating backbone network capacity, but it also reflects the pricing power of concentrated infrastructure. Operations that require large volumes of data to move between compute and the systems consuming inference results face materially different economics depending on network topology.

Shorter network paths, fewer intermediate operators, and direct peering relationships reduce both the latency and the cost of data movement. Infrastructure positioned at or near major Internet exchange points, where many networks peer directly rather than routing through intermediaries, benefits from lower effective bandwidth cost and more predictable latency.

Coastal metropolitan areas host some of the most densely interconnected Internet exchange infrastructure in the world. Compute deployed in these areas benefits from direct peering relationships that both reduce cost and improve performance compared to compute deployed in interior locations that must route through fewer peering points.

GPU Token Speed vs. Delivered Experience

Accelerator manufacturers publish performance benchmarks in tokens per second, floating-point operations per second, and related metrics. These metrics describe the raw capability of the accelerator in isolation. They do not describe the rate at which useful outputs are delivered to the systems consuming them.

The gap between accelerator throughput and delivered throughput is real and grows with network distance. An accelerator generating 10,000 tokens per second delivers those tokens to a user 60 milliseconds away at effectively 10,000 tokens per second. It delivers them to a user 2 milliseconds away at the same rate. The difference is in the time to first token, the latency before any output begins arriving, which directly affects perceived responsiveness.

For streaming inference, where output tokens are delivered to the user as they are generated rather than batched and sent at completion, the geography of the compute relative to the user determines the minimum time-to-first-token. A network path with 30 milliseconds of round-trip latency means the first token cannot arrive until at least 15 milliseconds after the request reaches the accelerator. Compressed further, the time-to-first-token is a directly user-visible metric for conversational and interactive applications.

Delivered Throughput and Effective Capacity

Effective capacity, the throughput that a compute deployment can actually deliver to users and systems, is a function of both raw accelerator performance and network delivery efficiency. Two deployments with identical accelerator specifications but different network positions will deliver different effective capacity to users in a given region.

This has direct implications for capacity planning. An organization that contracts for 100 gigaflops of inference capacity in a facility 80 milliseconds from its primary user base receives less effective capacity for latency-sensitive workloads than an organization contracting for the same specification 10 milliseconds from its user base. The hardware specification is identical; the delivered performance is not.

Effective capacity is also affected by the number of network hops and the reliability of each hop. Paths with more intermediate nodes have more points of potential congestion and failure. Paths that traverse fewer providers and fewer exchange points have more predictable performance.

Two deployments with identical accelerator specifications but different network positions will deliver different effective capacity to users in a given region.

Local Interconnection and IX Proximity

Internet exchange points, where multiple networks connect and exchange traffic directly, are concentrated in major metropolitan areas. Proximity to an IX allows a compute facility to peer directly with a large number of networks, reducing the number of hops between the facility and users on those networks and improving both latency and throughput.

IX proximity also provides resilience. A facility that peers with dozens of networks at a local IX has multiple redundant paths to its users. If one network experiences congestion or outage, traffic can re-route through other peers with minimal latency impact. A facility that connects to users primarily through a small number of transit providers has fewer options when one of those providers experiences problems.

Coastal metropolitan areas are home to major IX infrastructure in every major market. Compute positioned in these areas benefits from IX proximity as a structural advantage, not just a current feature.

The Subsea Infrastructure Advantage

Submarine fiber cables land at coastal points and connect directly to terrestrial networks at cable landing stations. These stations are among the most densely connected points in global Internet topology: many submarine cables terminate at a small number of landing stations, and those stations are connected to terrestrial networks via high-capacity fiber.

Compute infrastructure positioned near submarine cable landing stations benefits from proximity to the intersection of international and domestic network topologies. International traffic arriving via submarine cable does not need to traverse additional land distance before reaching the compute facility. Domestic traffic from the surrounding metropolitan area travels a short distance. The landing station is a network hub, and nearby compute inherits that advantage.

Seabase's subsea deployment model positions compute in and around the same coastal zones where submarine cables land. This is not incidental: the connectivity advantage of coastal cable proximity is one of the structural arguments for near-shore compute deployment.

Cold Water, Power Efficiency, and Operational Cost

Deep seawater is cold. At depths below a few hundred meters in most ocean regions, ambient temperatures are in the range of 2 to 8 degrees Celsius year-round. This thermal resource is available consistently, independent of surface weather, seasonal variation, or regional heat events that affect terrestrial data center cooling.

The power usage effectiveness, the ratio of total facility power to compute power, for a facility using deep cold water for cooling approaches 1.0 asymptotically: nearly all power consumed by the facility goes to compute rather than to cooling. Terrestrial facilities in warm climates, or those relying on air cooling, have PUE values significantly above 1.0, meaning a fraction of their power budget is consumed by the cooling system rather than by compute.

Lower PUE translates to lower operational cost per unit of compute delivered and to lower carbon intensity per unit of compute for facilities sourcing power from the grid. The cooling advantage of subsea deployment is not merely a technical curiosity: it affects the economics and environmental profile of the compute delivered.

The combination of network proximity, IX connectivity, and cooling efficiency creates a compound advantage for coastal subsea compute. No single factor determines the outcome; the interaction of all three is what makes the geography compelling. See the infrastructure overview for the technical architecture that realizes these advantages.

Nori and Regional Infrastructure Management

Operating compute infrastructure in multiple coastal regions requires tooling that reflects the distributed, persistent nature of the deployment. Capacity reservations, workload deployment, and monitoring must span multiple sites while providing a coherent operational view.

Nori by Seabase is the control plane through which Seabase customers manage their regional infrastructure. It provides visibility into capacity utilization across clusters, tools for deploying and managing inference workloads, and support and billing management in a single interface. Its design reflects the multi-region, persistent-allocation model: customers see their reserved capacity across all sites, not a pool of undifferentiated resources.

For organizations operating latency-sensitive AI infrastructure, the ability to see and manage capacity at each regional site, and to understand which workloads are deployed where, is operationally important. Nori provides that visibility without abstracting away the geographic detail that matters for latency-sensitive applications.

Where Orbital and Satellite Infrastructure Fits

The argument that LEO satellite connectivity resolves AI infrastructure geography is most compelling for specific use cases where it is genuinely true. Maritime operations, aviation, remote sensing, Earth observation, and rural or wilderness coverage all benefit significantly from LEO connectivity and, in some cases, from compute positioned at or near LEO ground stations.

Space-based applications represent a distinct and growing category. Satellite constellations that process imagery, perform onboard inference, or coordinate with ground systems benefit from compute positioned close to ground station infrastructure. The NASA constellation management programs and commercial satellite operators face real latency constraints that coastal and orbital infrastructure together can address.

The key distinction is that these use cases are defined by the absence of terrestrial alternatives, not by the superiority of satellite delivery over terrestrial fiber for users who have both options. The appropriate infrastructure strategy treats LEO satellite and coastal terrestrial compute as complementary rather than competing: satellite for coverage extension, coastal infrastructure for dense demand centers.

For workloads that cross both categories, a multi-tier architecture where edge compute handles real-time inference and regional coastal clusters handle larger models and longer-horizon tasks, the geography of each tier should be matched to its latency requirements.

Geography as a Durable Infrastructure Advantage

Network improvements over time reduce but do not eliminate latency. The speed of light is a constant. Protocol overhead decreases as protocols improve but does not reach zero. The latency difference between 5 milliseconds and 50 milliseconds is durable: no amount of software optimization changes the physics.

This means that infrastructure positioned close to demand has a durable performance advantage over infrastructure positioned far from demand for latency-sensitive workloads. The advantage does not erode as networks improve; it persists as long as the physics holds.

For infrastructure operators, geographic positioning is therefore a long-duration asset. The capital investment in placing compute close to a major metropolitan area creates an advantage that compounds as AI workloads become more latency-sensitive and more numerous. A facility built far from demand does not become competitive by improving its network; it becomes competitive only by adding infrastructure closer to demand.

The organizations that will operate the most valuable AI infrastructure in ten years are those building it in the right places today. The latency tax is not a transitional problem that better technology will eventually solve. It is a structural feature of the physics of information, and it rewards infrastructure operators who take geography seriously.

The latency tax is not a transitional problem that better technology will eventually solve. It is a structural feature of the physics of information.

Building for the Geography That Exists

The premise that geography would stop mattering for compute was always conditional on workload type. For batch processing, asynchronous analytics, and human-paced conversational AI, the condition has been met: remote data centers deliver acceptable performance. For real-time AI, physical automation, and persistent inference at scale, the condition has not been met and will not be.

The infrastructure investments being made now will serve workloads for a decade or more. The organizations planning those investments should be asking not only how much compute they need, but where it needs to be. The answer, for a growing share of AI workloads, is near demand, and near demand means coastal.

For the full case on why persistent workloads require regional infrastructure, see Beyond Chatbots: The Case for Persistent Regional AI. For the environmental accountability framework governing subsea deployment, see Environmental Accountability for Subsea AI Infrastructure. To discuss regional infrastructure requirements, reserved capacity options, or network positioning, contact Seabase.