Glossary

AI Inference

AI inference is what happens when a trained model is used in the real world. It turns a prompt, image, sensor reading, or other input into an answer or action.

· 4 min read

← Glossary · Articles · Pillar guide

Definition

AI inference is the process of using a trained AI model to generate an output from a new input. Training teaches a model patterns from data. Inference applies those learned patterns to live work such as answering a question, summarizing a document, classifying an image, generating code, translating speech, or ranking results.

When someone sends a prompt to a language model and receives a response, that response is produced through inference. The model is usually not being retrained during that exchange. It is performing calculations with existing weights to estimate a useful output.

Why it matters

Inference is where AI becomes an operating service. A model can be impressive in a lab, but it only becomes useful at scale when it can answer many requests reliably, quickly, and at a manageable cost.

For many applications the important constraints are latency, throughput, availability, and cost per request. As AI adoption grows, inference can represent a large share of ongoing compute demand. Training may be a major event. Serving millions of requests is an ongoing operational workload.

How it works

Input data is converted into a numerical form the model can process. Text becomes tokens. Images become arrays. The model then performs layers of mathematical operations, often on AI accelerators, to produce a result.

Inference systems usually include request routing, batching, memory management, caching, monitoring, and safety controls. Large models may be split across multiple accelerators connected by high-speed networks. Smaller or time-sensitive models may run closer to users so data does not travel as far.

Common misconceptions

Inference is not automatically simple because training already happened. Large-scale inference must keep expensive hardware busy while meeting user-facing latency targets.

Not all inference should happen in one centralized location. Centralization can help utilization, but distance adds delay. Some workloads benefit from regional or metro placement near users and data.

See The Latency Tax, Why AI Inference Changes Infrastructure Geography, Persistent Regional AI, and Metro AI Infrastructure in the glossary.