Inference

Inference is the act of running a trained model on new input to get a prediction or output. When you send a prompt to GPT-4 and get a response, that is inference. Training builds the model; inference uses it. The distinction matters because the economics are completely different. Training a frontier model costs tens of millions of dollars and happens once (or a few times). Inference happens millions of times per day and is where the ongoing cost lives. Inference speed—measured in tokens per second—determines whether your AI feature feels instant or sluggish. Inference cost—measured in dollars per million tokens—determines whether your AI feature is economically viable at scale. Much of the current hardware race (custom chips from Google, Amazon, startups like Groq and Cerebras) is about making inference cheaper and faster, not training bigger models.