How to think about AI latency in product design

When teams first integrate an LLM into a product, latency is usually treated as an infrastructure problem. The AI call takes two seconds; the goal is to get it to one second. If you can get it under a second, the problem is solved. This framing is not wrong exactly, but it is incomplete in ways that matter for product quality.

AI latency is not a single number. It is not symmetric. It does not behave like traditional API latency, and the design decisions that work for traditional API response times often produce bad experiences with AI. Understanding why requires thinking about what latency actually is from a user’s perspective, not just from an infrastructure perspective.

The difference between latency and perceived latency

Traditional API latency is experienced as a delay between action and response. The user clicks, something spins, the result appears. The interval between click and result is the latency. Minimizing that interval minimizes the experience of waiting.

AI latency does not work this way, for two reasons.

First, AI responses are often long. A traditional API call returns a result; an AI call returns text that takes time to read. If the first token of a long response arrives quickly, the user starts reading while the rest of the response generates. The experience is reading, not waiting. This is why time to first token is often the metric that matters most for interactive AI: once the user is reading, the generation time is largely absorbed by the reading time.

Second, AI output is often uncertain. With traditional APIs, the user knows roughly what the result will look like and is waiting for it to arrive. With AI output, users often do not know what is coming. An apparent delay before content appears is experienced differently depending on whether the user expects a short answer or a long one, whether the task is time-sensitive, and whether the delay is explained by visible progress indicators.

This means two AI products with identical p99 latency can produce very different user experiences, depending on how they handle streaming, what they show during generation, and what users expect.

What makes AI latency feel acceptable

The research on user tolerance for latency is well-established for traditional interfaces: under 100ms feels instantaneous, under 1 second feels immediate, under 10 seconds keeps attention, beyond 10 seconds requires explicit progress feedback. These thresholds transfer poorly to AI.

For AI interactions, the relevant factors are different.

Does the user have something to do while waiting? If the AI response streams into a readable area, users start reading while generation continues. This fundamentally changes the experience: generation time is not wait time, it is reading time. A product that streams a 1000-word response over 8 seconds while the user is actively reading feels faster than a product that returns a 100-word response after a 3-second delay.

Does the user understand why it is taking time? AI generation involves processes that users cannot observe. A visible indicator that generation is happening, and ideally some signal of progress, makes the wait feel more acceptable. The absence of visible progress turns uncertain waits into anxiety. The classic failure mode is a blank loading state with no content until the full response appears: users cannot distinguish this from the AI being broken.

Is the latency consistent? Variable latency is experienced worse than consistent latency at the same average. A product that sometimes responds in 1 second and sometimes in 8 seconds creates uncertainty that makes every response feel potentially slow. A product that consistently responds in 3 seconds is experienced as being slower than it is, but at least it is predictable. Users adapt to consistent latency; they cannot adapt to variable latency.

Is the task latency-sensitive? Latency tolerance is context-dependent. A user asking the AI to summarize a document they will read later tolerates more latency than a user mid-conversation asking a follow-up question. A user using the AI to draft a response they will send shortly has different latency requirements than a user using the AI to generate content they will review tomorrow. Product design that treats all AI calls as equally latency-sensitive misallocates effort.

The constraints that cannot be engineered away

There are real lower bounds on AI latency that engineering cannot fully eliminate. Understanding them is important for setting appropriate expectations and making correct product decisions.

Model size and capability. More capable models are generally slower. A product that requires a capable model for quality reasons cannot achieve the latency of a product using a smaller, faster model. The decision to use a more capable model is a decision to accept higher latency, and product design needs to account for that.

Input length. AI inference time scales with the length of the context being processed. Products that pass long documents, conversation histories, or complex system prompts to the AI will see higher latency than products with short, focused prompts. Reducing input length is often one of the most effective latency improvements available, but it requires making deliberate decisions about what context the AI actually needs.

Output length. Longer responses take longer to generate. If the use case requires long responses, generation time will be proportionally higher. Structuring outputs to put the most important content first (so users can start acting on the response before generation completes) is a design decision that affects perceived latency without affecting actual generation time.

Network and infrastructure. AI inference often happens on infrastructure that the product team does not control. Network latency to inference providers, provider queue depths, and cold start times for serverless inference all affect latency in ways that are partially outside engineering control.

What to do about it

Given these constraints, the design question is not only “how do we make AI faster” but also “how do we design so that AI latency does not feel like a problem.”

Stream everything. If you are building an interactive AI product and you are not streaming responses, start there. Streaming reduces the most important latency metric (time to first content) and converts generation time from wait time to read time for users. The implementation cost is low; the experience improvement is often significant.

Show progress, not absence. Design states for AI generation that give users something to observe. This can be as simple as showing that generation is in progress, or as detailed as showing intermediate outputs as they are generated. What you cannot do is show nothing and expect users to trust the system.

Design for the worst case, not the median. AI latency has a long tail. Your median response may come back in two seconds; your p95 may be eight seconds. Product design that assumes median latency will produce bad experiences for the tail. Design for the realistic worst case: what should the user see if this takes ten seconds?

Distinguish latency-sensitive from latency-tolerant use cases. Not every AI call needs to be fast. Batch processing, background generation, and async workflows can tolerate much higher latency than interactive conversations. Separating these architecturally lets you invest in latency reduction where it matters and accept latency where it does not.

Reduce unnecessary context. Audit what is actually being passed to the AI in production. Long system prompts with content the AI never uses, full conversation histories when only the last few turns are relevant, and complete documents when the user’s question requires only a section: all of these increase latency without improving output quality. Focused context is often faster context.

The product framing

The right way to think about AI latency in product design is not as a number to minimize but as a constraint to design around. Some of that constraint can be reduced by engineering. Some of it is fixed by the capabilities you need. The design question is how to build a product that works well within the constraints that remain.

The teams that get this right start from what users actually experience rather than from infrastructure metrics. They distinguish between latency that users perceive and latency that instruments measure. They design streaming behavior, progress states, and user expectations as deliberately as they design features. And they make deliberate tradeoffs between model capability, context length, and response latency rather than treating all three as independent variables to optimize separately.

AI latency will improve over time as model efficiency improves, inference infrastructure matures, and the ecosystem around AI applications develops. In the meantime, the products that feel fast are usually not the ones with the lowest measured latency. They are the ones designed by teams who understood what latency actually feels like to users.