How should an AI product ask for feedback on its answers, and what should happen next?

Illustrative situation: a founder adds thumbs-up and thumbs-down buttons under every answer in a document assistant built on a hosted language model. A month later the dashboard shows 90% thumbs-up, and only a small share of answers got a click at all. Support keeps hearing about the same three problems, and none of them shows up in the thumbs. The founder cannot say whether the product is getting better. The scene is invented, but the pattern is common: the buttons were treated as a measurement when they are one weak signal among several, and nobody decided what happens after a click.

Humbleteam, a digital product design agency, publishes this guide. It treats feedback as a loop with four parts: what you capture, when you ask, who reads it, and what the user sees back. A signal map near the end puts all four on one page, followed by an invented worked example.

What can a product learn from each kind of feedback?

Google’s People + AI Guidebook separates feedback into two kinds. Implicit feedback is data about behavior in your product logs, such as how often people accept or reject a recommendation. Explicit feedback is deliberate commentary on output: ratings, thumbs, surveys, open text. For both, the guidebook says to tell people what is collected and what it is for, let them opt out of behavior logging, and give them a way to see the data.

Neither kind is a clean readout. The guidebook warns that interacting with something does not mean wanting more of it, because people may be trying to dismiss it or following a passing curiosity. In chat products, Eugene Yan points out that a longer conversation could mean engagement or a struggle to get an answer. Langfuse’s documentation lists the trade: explicit feedback is clear but sparse and skewed toward unhappy users, while implicit feedback is plentiful but ambiguous.

Some implicit signals are stronger than they look. In a study of GitHub Copilot users, the rate at which shown suggestions were accepted, rather than how long completions persisted in the code, was what reflected developers’ perception of productivity. In a company study of a coding chat assistant used by 372 developers over nine weeks, thumbs appeared on 0.6% of turns, while sentiment in the developers’ own follow-up prompts gave a usable signal in about 8% of interactions. The narrow lesson: the next message a user types is a feedback channel you already have.

Where and when should the product ask?

NN/g’s rule for feedback requests is “task, then ask.” Don’t interrupt before people have done anything, keep the request from covering information they need, and add an on-demand feedback entry for people who dismissed the first prompt. For an AI product that gives three placements:

• Per answer: right for short exchanges where each answer is the task, such as a chat reply. Put the thumbs under the answer.
• Per task: right when the answer is one step in a workflow, such as summarize, edit, export. Ask after the user finishes the step.
• Sampled: Microsoft’s pattern 15B has the system occasionally ask for explicit feedback on selected outputs. Teams decide how often to ask and whether to ask all users or groups on a rotating basis. The pattern’s own pitfall list includes asking too often and a sampling method that introduces bias.

Visibility matters too. In a May 2023 survey of early ChatGPT users, the most common reason for not submitting feedback, given by 27 of the 78 respondents who hadn’t, was not knowing the option existed. The sample is small and self-selected, but the point carries: a control users do not notice collects no feedback.

How do you design the reason step without friction?

NN/g notes that closed-ended questions are easier to answer than open-ended ones, and recommends an optional free-text field at the end with a character limit. Its example is an expense app that showed a thumbs-up or thumbs-down after the user finished a task, plus an optional 500-character box. It also says to label clearly which options are positive and which are negative.

Our suggested sequence for an AI answer: one tap on the thumb, then three to five optional reason chips after a thumbs-down, then an optional text box. A bare thumb still counts. Nothing is required, and nothing opens a full-screen form.

Chip wording is easy to get wrong by guessing. The People + AI Guidebook asks for options that are mutually exclusive and collectively exhaustive, matched to how users judge the experience. Build them from your own failures. Hamel Husain’s field guide to improving AI products describes a bottom-up method: read real conversations, write open notes on what went wrong, group the notes, and count. In his example from an apartment-industry assistant, three problem types made up over 60% of all issues. Those groups, in the users’ words, become your chips. Google’s Gemini app shows the pattern in a shipped product: after a rating, you can optionally select a reason, and Google says a reason helps it understand the rating.

Is an edit a label?

Often it is. When someone changes generated text, they are telling you where the output fell short. A NeurIPS 2024 paper on user edits treats edits as feedback that is generated naturally and can reveal latent preferences.

An edit can also mean success. Yan counts “accepting and making minor tweaks” as positive feedback and wholesale acceptance as strongly positive. Both readings hold, so size and type matter. Classify each edited answer as unedited, small wording edit, factual change, or rewrite. Store the type and the span that changed. Sample the factual changes for review. For the editing controls themselves, such as selection, versions and undo, see our guide to editing AI-generated work without starting over.

What should the user see after giving feedback?

The People + AI Guidebook lists five confirmation messages, from “Thanks for your feedback” (no timing, no scope) to “We’ve updated your recommendations. Take a look” (immediate, shown). It advises acknowledging feedback and adjusting right away or saying when, and promising only what the product can deliver on that timeline. Microsoft’s guideline 16 says to immediately update or convey how user actions will affect the system’s future behavior.

Follow-up is easy to skip. In the ChatGPT survey above, 11.2% of the 152 people who had submitted open-text feedback said they received any follow-up or update. Write the message for what actually happens:

• If a person reads reports weekly: “Thanks. Our team reads these reports.”
• If the report becomes a test case: “Saved. We use reports like this to check future changes.”
• If a fix ships: a release note that names the report: “You told us the summary missed a renewal date. It now flags dates for review.”
• If the user can get a better answer now: regenerate with their reason as an instruction, and show it.

Never write “your feedback trains the model” unless a training step exists. The next two sections explain why.

What do model providers do with feedback, and what do you owe your users?

Providers publish their own rules, and they differ by plan. As of October 27, 2026:

• OpenAI. For Free, Plus and Pro, a thumbs rating means the entire conversation may be used to train its models, even after opting out of training. For Business, Enterprise, Edu and the API, inputs and outputs aren’t used to improve models by default, and thumbs feedback in business workspaces doesn’t permit training. API organization owners can turn data sharing on.
• Anthropic. On consumer plans, a thumbs rating stores the entire conversation for up to five years, de-linked from the user ID, and it may be used to train models. For commercial products and the API, inputs and outputs aren’t used for training by default, but explicitly reported feedback can change that. Team and Enterprise owners can disable the thumbs button.
• Google. In the Gemini app, feedback and its associated conversation are reviewed by specially trained teams and kept up to three years, disconnected from the account. For the Gemini API, paid services don’t use prompts or responses to improve Google products, while unpaid services do and may route them to human reviewers.

Two things follow. These buttons are the providers’ feedback channels in their own apps. Your product’s thumbs go to your database, and the provider’s API terms govern what you send it. And your users need their own notice. Under the EU’s GDPR, personal data must be collected for specified, explicit and legitimate purposes and not processed in an incompatible way. This is general information, not legal advice. In a B2B product, feedback often contains customers’ documents, so log metadata by default and ask before copying full conversations into evaluation sets.

How does feedback reach evaluation sets and review queues?

Treat a click as a pointer to a conversation worth reading, not as a verdict. Langfuse records feedback as scores linked to traces, which you can filter, send to annotation queues or use as ground truth for automated evaluations. LangSmith can attach user feedback to any step of a trace and offers annotation queues where human reviewers work through runs or whole threads. For reviewers, Husain recommends a binary pass or fail plus a short written critique over a 1-5 scale.

Our routing rule: fill the weekly queue from three streams. First, every thumbs-down and reason chip. Second, factual-change edits and regenerations. Third, a random sample of unrated answers, so you can see what silence hides. A reviewer marks each item pass or fail with a one-line critique. Confirmed failures become evaluation cases with the input, the expected result and the failing field, and you rerun them on every prompt or model change. Then say what the feedback changed: a prompt, a retrieval rule, a test, a screen.

Be careful with any training use. OpenAI’s own account of its April 2025 sycophancy problem says a new reward signal built from thumbs data was “often useful,” but that together with other changes it weakened the primary signal that had been holding sycophancy in check. User feedback, it said, can sometimes favor more agreeable responses, and its offline evaluations and A/B tests did not catch the problem. Its first post said the update focused too much on short-term feedback. If you ever tune on thumbs, keep separate checks for the behavior you refuse to trade away.

Who clicks, and how should you read a low response rate?

Jakob Nielsen’s participation inequality describes online communities, not AI products, but the mechanism travels: in most communities 90% never contribute, 9% contribute a little, and 1% supply most of the contributions, so what you hear comes from an unrepresentative few. Microsoft’s pattern 15B names reporting bias, where users with moderate views don’t give feedback, and says to complement the data with other sources. Langfuse adds that unhappy users are more likely to respond. In the coding-assistant study, the average sentiment of follow-up prompts was negative; the authors suggest that refining a weak answer takes a message while praising a good one often doesn’t. Rewards don’t solve it: the People + AI Guidebook lists cash as a way to raise volume that may bias toward a subset of users and lower quality.

How to read what you have:

• A 2% response rate tells you about the 2%. It says nothing about the other 98%.
• The share of thumbs-down among raters is not your failure rate.
• Compare who rates with who uses, by plan, role and task.
• Track the response rate itself. A new button position or a new moment to ask can change it with no change in answer quality.
• Fill the gap with implicit signals and with reading unrated answers.

The feedback signal map

Fill one row per signal your product can capture. The reliability column is our reading of the sources above, not a measured figure.

Illustrative worked example: a contract-summary tool

Every number here is invented. A 14-person startup sells a tool that summarizes vendor contracts for procurement teams. It runs on a hosted language model. Over one month the team records this:

Read alone, the thumbs look healthy: 85% of the 400 raters clicked up. The edit count says otherwise. The team reads 100 edited summaries and the 60 thumbs-down. Of the 100 edits, 41 fix a date, amount or party name, 24 delete a sentence the contract does not support, 22 shorten or restyle, and 13 are other. Of the 60 thumbs-down, 21 are about the export layout, not the summary.

Two conclusions follow. About 25% of all summaries (60% edited, times 41%) may need a factual fix, with wide uncertainty because the sample is 100. And the thumbs sit under the summary but are being used to complain about the export screen.

The team makes five changes:

1. It logs each edit’s type and the changed span, not the contract text, because customers’ contracts are confidential.
2. It builds a weekly queue: 30 factual-change edits, every thumbs-down, and 10 unedited summaries chosen at random. One reviewer marks pass or fail with a critique.
3. It turns the confirmed date and amount failures into 25 evaluation cases that rerun whenever the prompt or model changes.
4. It proposes a check that compares extracted dates, amounts and party names with the clause text, and highlights those fields for the user.
5. It moves the thumbs prompt to after export, adds a separate “report a problem with the export” link, and posts a short note in the app each month: “You corrected dates and amounts most often. Here is what changed.”

Next month the team watches the share of summaries with factual-change edits, the response rate of the moved thumbs, and the unrated sample. It makes no claim yet about improvement, because it has not measured one.

Where do these rules stop working?

The sources mix design guidance, vendor documentation and a few small studies of particular products. They show what practitioners recommend, not what your users will do. Provider terms change, so read the current pages before you rely on them. This guide does not cover voice products, safety reporting flows, or how to train a model on feedback.

What can you produce now?

Five things, before anyone redraws a screen:

1. A signal map for your product, with an owner for each row.
2. A reason step: where the thumb sits, which chips appear, and what stays optional.
3. A weekly review ritual with its three streams, a pass or fail rule, and a place where confirmed failures become tests.
4. A promise sentence for each outcome, written to match what really happens.
5. A data-use note that says which feedback is stored, who sees it, and what it is not used for.

If you are choosing an outside team to design these flows, our comparison of agencies that have designed LLM-powered products for startups shows which studios publish cases on how people check or correct AI output.