How should an AI product design the wait for a response?

Illustrative situation: a founder ships an assistant on top of a hosted language model. In demos it answers in a few seconds. In production some requests take twenty seconds, a few take minutes, and now and then one never comes back. The screen shows the same spinner for all of them. Users click Send twice, reload the page or leave, and support hears “it froze” while the logs say the model was working. The scene is invented, but the gap is common: the team measured how long the model took, not what the user saw, could do and was told during that time.

Humbleteam, a digital product design agency, publishes this guide. It treats the wait as a designed state with four parts: how long it lasts, what the screen shows, what the user can do, and what happens when it fails. A worksheet near the end turns the four parts into a table you can fill in for your own product, followed by an invented worked example.

Which time limits still apply to AI products?

NN/g’s response-time guidance goes back to Jakob Nielsen’s 1993 book, with a 2014 update, and it names three limits. About 0.1 second feels instantaneous, so no special feedback is needed. About 1 second keeps the user’s flow of thought unbroken, though the delay is noticed. About 10 seconds is the limit for keeping attention on the dialogue. The 2014 update adds that anything slower than 10 seconds needs a percent-done indicator and a clearly signposted way to interrupt, and that users need to reorient when they return after such a delay. The limits come from human perception, not technology, which is why they still describe a language model call.

Indicator guidance builds on them. NN/g’s article on progress indicators says to give immediate feedback for every action, use an indicator for anything over about one second, use looped animation for waits of 2 to 10 seconds, and use a percent-done indicator at 10 seconds or more. Material Design 3 draws the lines elsewhere: no indicator under 200 milliseconds, a loading indicator up to 5 seconds, a progress indicator above that. Two respected systems disagree on the cut-offs, so pick yours and write them down.

Faster is not automatically read as better. A CHI 2026 experiment with 240 US participants fixed the time to first token at 2, 9 or 20 seconds, then streamed the text at the same pace, with no progress cues. Behavior in the logs stayed steady across the groups. People who waited 2 seconds rated the answers less thoughtful and useful than those who waited 9 or 20 seconds. At 20 seconds, one participant said the delay made them question whether their internet had cut off. The authors call latency “a tunable design variable with ethical implications.” Our reading: do not pad responses with fake pauses. The study left cues out on purpose, and a cue is the cheap fix for that doubt.

Which number should you track: time to first token or total time?

Anthropic’s latency guide defines time to first token (TTFT) as the time from sending the prompt to the first token of the response, and says it matters most when you stream. AWS’s Agentic AI Lens says user-facing agents are judged on perceived latency, calls TTFT the dominant signal, and asks teams to track it as its own KPI. It also names a trap: streaming the model call while retrieval and prompt assembly still add seconds before anything appears.

OpenAI’s latency guide calls streaming “the single most effective approach” because it cuts the waiting time to a second or less, and adds that streaming reduces latency for the app and the user together, since people finish reading sooner.

Our suggestion is three clocks per request type. Acknowledge time runs from the click to the first visible change on screen. First-result time runs to the first token or the first usable piece of output. Done time runs to completion. Record the median and the 95th percentile for each, because users remember the slow runs.

What does streaming fix, and where does it break?

Streaming starts the reading while the model is still writing. It also creates problems a finished answer never had.

• Scroll fights the reader. Many chat views stay pinned to the bottom, so scrolling up to reread something snaps the page back down. Smashing Magazine’s guide to stable streaming interfaces suggests following the stream only while the user is at the bottom, stopping when they scroll up, and resuming when they return. The same guide covers a polite aria-live region for screen readers and a reduced-motion setting for the typewriter effect.
• Layout shifts. As the message grows, everything below it moves, including a button the user was about to press. Keeping the action row in a fixed place until the answer is done is our suggestion.
• Half an answer can be withdrawn. OpenAI’s streaming guide warns that partial completions are harder to moderate, and that moderation scores arrive only after the full output. Anthropic’s streaming refusals page shows a response that stops with partial text and a “refusal” stop reason. Decide in advance what the interface does with an answer that ends mid-sentence for a reason the user cannot see.
• Structured results are unreadable halfway. Anthropic says structured outputs stream like normal responses, but tool input arrives as partial JSON strings that are parsed at the end. Current models emit one complete key and value at a time, so delays between events can occur. With fine-grained tool streaming you may receive partial or invalid JSON. The limit sits in the interface: a table, chart or form is useless until it is whole. Stream prose, and reveal structure in one piece once it validates.
• Silent pauses. AWS lists an unexplained stall while the agent calls a tool as an anti-pattern. A short status line fixes it.

Skeleton, spinner, progress bar or step list?

NN/g’s skeleton screen guidance is specific. Skeletons suit full-page loads that take under 10 seconds. Spinners suit a single module and waits of 2 to 10 seconds. Progress bars are strongly recommended above 10 seconds. A frame with no content outline is not recommended. The progress article adds that a static “Loading…” should go, and so should any “don’t click again” warning, because the fix is to show that the first click was accepted.

For an AI product, that becomes a simple rule. Use a skeleton only when the shape of the result is known before it arrives, such as a brief with fixed sections. For an open-ended answer, show a working label until the first token, then the stream. For a multi-step run, show steps.

Steps suit agents because their duration is unknown. Nielsen’s essay covers this case: when the total work is not known, report the absolute amount done, such as naming each database as it is searched. OpenAI’s guide says the same of tools: if the product takes multiple steps or uses tools, surface them, and “the more real progress you can show, the better.” Apple’s progress indicator guidance adds that labels should be accurate, since vague words like “loading” rarely add value. Material says to use one indicator for a group of items, not one per activity.

What should people be able to do while they wait?

They should be able to stop. Nielsen asks for a signposted interruption above 10 seconds. Apple says to include a Cancel button when interrupting causes no harm, add Pause when stopping would lose work, and tell people when halting has a cost. A stop button also has to leave a clean state. Smashing’s guide lists the pieces: clear the pending buffer, remove the blinking cursor, mark the response as incomplete, reset the buttons. We would keep the text already received, label it “Stopped,” and offer Continue and Regenerate. If the user regenerates, keep the earlier version reachable until they pick one.

They should also be able to leave. Material suggests that, for a very long wait, people can navigate away while the process finishes. OpenAI’s background mode exists for this: it starts long tasks asynchronously, and the developer polls for status. Its webhooks can notify your server when a background response is generated. NN/g’s progress article gives a human parallel in a phone queue that offered to call back and keep the caller’s place, which freed the caller to do other things.

Two cautions, as of October 28, 2026. OpenAI’s docs say the time to first token of a background response is currently higher than a synchronous one, so reserve background runs for work that justifies it. They also say a background response is deleted after roughly 10 minutes when store is omitted or set to false, so copy results into your own database before promising “we’ll tell you when it’s ready.” A notification that opens a missing result is worse than none. Offer “keep working in the background” once a run passes a threshold you choose, and keep a run list that shows status.

What happens on a timeout, an error or a partial result?

Time limits sit at several layers: the browser connection, your server and the provider. Anthropic notes that its SDK requires streaming for requests with large output limits to avoid HTTP timeouts, and that the API can send errors inside the event stream, for example an overloaded error during busy periods. It documents a capture-and-continue pattern for broken streams: save what arrived and send it back with an instruction to continue. Text can resume that way. Tool-use and thinking blocks cannot be partially recovered, so a product may be able to continue a lost answer but not a lost action.

Decide the timeout for each request type, then design the message. Google’s People + AI Guidebook says that after a failure the easiest path is often to let the user take over, and that you must make it easy to pick up where the system leaves off. Microsoft’s HAX guideline 9 says AI systems will sometimes be only partially correct, so outputs should be easy to edit and recover.

Example wording, ours: “This took longer than 6 minutes, so we stopped it. We kept the 9 pages it had read and the notes so far. Continue, start over with fewer sources, or open the notes.” It states the limit, says what is saved and gives three exits. “Something went wrong” gives none.

How do you keep progress honest?

Apple puts the risk plainly: showing 90 percent completion in five seconds and the last 10 percent in five minutes can make people wonder if the app is still working and can feel deceptive. A stationary indicator reads as a stall, so explain the problem. Material says a determinate indicator must accurately represent what it measures, and should switch from indeterminate to determinate as more becomes known. NN/g warns that a bar hanging on the last percent cancels the benefit of showing progress, and advises a general time estimate or a step count over a number that will turn out wrong.

On pace the sources differ. NN/g suggests starting an animation slower so it does not set an expectation the system cannot keep, while Apple suggests evening out the pace. We read both as advice about a bar that tracks work that is really happening, not permission for a bar driven by a timer.

For model-driven runs, the total number of pages, tool calls or tokens is usually unknown at the start, so a percentage would be a guess. Our rules:

1. Show a percent only when the denominator is real, such as files uploaded or pages in a document you are processing.
2. Otherwise show steps and counts of work done, plus elapsed time.
3. Give an estimate as a range from your own median and 95th percentile: “usually 1 to 4 minutes.”
4. When a run passes its 95th percentile, say so, and offer to stop or continue in the background.

A latency budget worksheet

Fill one row per request type, and add your measured median and 95th percentile to each. The bands are our starting suggestions built on the limits above, not measured figures.

Illustrative worked example: an account-research assistant

Every number here is invented. A nine-person startup sells an assistant that drafts one-page briefs on target companies from public web pages. In one week the team records: quick questions reach a first token at a median of 1.4 seconds and a 95th percentile of 7 seconds. Briefs finish at a median of 85 seconds and a 95th percentile of 4 minutes, and 3 percent are still running at 6 minutes. The screen has one spinner and a disabled Send button, with no stop control.

The team makes six changes:

1. The user’s message appears and Send becomes Stop within the first moment of the click.
2. Quick questions stream. Because the 95th percentile is 7 seconds, a “Reading your question” label covers the gap before the first token.
3. Briefs have fixed sections, so the screen shows a three-section skeleton for the first few seconds, then a step list: “Searching, 4 queries done,” “Reading pages, 7 read,” “Drafting.” No percentage appears, and the estimate says “usually 1 to 4 minutes.”
4. At 60 seconds, the run offers “Keep working, we’ll tell you when it’s ready.” The finished brief is stored in the team’s own database before the notification goes out.
5. Stop keeps the pages already read and the notes so far, labels the brief “Stopped,” and offers Continue.
6. At 6 minutes the run ends with the three-exit message from the timeout section.

Next month the team watches the 95th percentile of first-result time, the share of runs stopped, the share of background runs opened from a notification, and the rate of duplicate runs started within two minutes of another. It makes no claim yet about improvement, because it has not measured one.

Where do these rules stop working?

The sources mix perception research from 1993 and 2014, vendor documentation that changes, a practitioner article and one controlled study with US online participants and a single model. They show what practitioners recommend, not what your users will do. The CHI study varied only the time to first token and left out progress cues, so it says little about cues. Provider behavior, including background retention and recovery steps, was checked on October 28, 2026 and can change. This guide does not cover voice products, token pricing or model choice.

What can you produce now?

Four things, before anyone redraws a screen:

1. A latency budget with one row per request type, filled with your own median and 95th percentile.
2. A list of the states each row needs: waiting, streaming, stopped, timed out, finished in the background.
3. A progress rule that says when a percent is allowed and what appears otherwise.
4. A timeout message for each row that states what is saved and what the user can do next.

If you are choosing an outside team to design these states, our comparison of agencies that have designed LLM-powered products for startups shows which studios publish cases on how people check or correct AI output. Two related guides cover what to show when the system is unsure, in when an AI product should ask, qualify or decline, and what to do with a click on thumbs-down, in how an AI product should ask for feedback.