How to evaluate a design agency when every pitch looks good
Three polished proposals can still leave a founder unsure whom to hire. Compare the evidence behind each pitch with a practical agency-evaluation worksheet.
Brand and Marketing Director
A founder finishes three agency calls with three polished portfolios, three confident proposals, and no clearer idea of whom to hire. Everyone promises senior people and a collaborative process. The prices differ, but so do the things included.
To evaluate a product design agency, compare evidence against the same product problem: a relevant past decision, the people assigned to the work, the assumptions behind the proposal, and what a former client can confirm. Record unanswered questions separately from reasons to reject a team.
This guide offers a worksheet for that comparison. Humbleteam publishes it and sells product design services; the same questions apply when evaluating Humbleteam.
What should every agency be evaluating?
Before comparing proposals, write one brief that every candidate receives. It should describe the user’s task, the observed difficulty, the next product decision, and the constraints the team already knows.
Consider a hypothetical B2B SaaS startup. New customers create accounts but need support to import their data. The founder initially asks for an onboarding redesign. Yet the difficulty might involve file formats, error recovery, unclear instructions, or something outside the interface.
A useful brief says: “Customers need to import their existing data and reach a usable workspace. Support currently intervenes during import. The first decision is which obstacles require interface changes and which require engineering work.”
Attach the available evidence, explain what remains unknown, and name the people available to help. If nobody can recruit customers or approve changes, the agency needs to know before proposing a schedule. The redesign baseline brief covers how to assemble this starting point.
What proves that a portfolio is relevant?
Choose one comparable project and ask the proposed delivery team to walk through a decision within it. A shared industry helps, but the practical constraint matters too. A small team shipping around a legacy data model may be a closer comparison than a famous company with an unlimited redesign brief.
Ask what the agency inherited, what it personally changed, and what reached production. Then ask which option it rejected and why. Screenshots can show the result; an explanation should connect that result to the evidence available at the time.
When a case study claims an improvement, ask what was measured, over which period, and what else changed. A conversion increase following a redesign does not, by itself, establish that the redesign caused it.
Confidentiality can limit what a team shares. A redacted walkthrough or a client-approved reference may provide enough evidence. “Unable to verify” deserves its own entry in the notes; it is not proof that the work never happened.
How can a founder compare proposals fairly?
Use the following worksheet for each shortlisted agency. The founder and engineering lead should fill it out independently before discussing their preferences.
| Decision | Evidence to request | Question to resolve |
|---|---|---|
| Does the team understand the problem? | Its restatement of the brief and initial questions | What would change the proposed scope? |
| Has it handled a relevant constraint? | One project walkthrough with a specific decision | Which parts did this team own? |
| Who will do the work? | Named contributors, responsibilities, and availability | Who handles reviews and absences? |
| Can the startup support the plan? | Customer access, data, and approvals required | What happens if a dependency is late? |
| Can engineering use the output? | A sample handoff and agreed implementation support | Who resolves missing states during development? |
| What does the price cover? | Included work, exclusions, and revision assumptions | Which events would change the estimate? |
Beside each row, record the evidence, its source, and a status: supported, unresolved, or unsuitable for this project. Add the next action for every unresolved item. A numerical average can hide the one condition that makes delivery impossible, such as the required designer being unavailable.
Lenny Rachitsky’s guide to interviewing product managers recommends individual judgments before the panel discussion. Applying that idea to agency selection gives each reviewer a chance to record concerns before hearing the founder’s favorite. It is an adaptation of a hiring practice, not a validated agency-ranking formula.
What should a reference call uncover?
A reference is another perspective, with its own limits. An agency usually introduces a satisfied client, and that client’s project may differ substantially from the proposed work.
Start by establishing what the reference directly observed: the scope, the people involved, and their own role. Then ask about specific events:
- When did the plan change, and how did the team handle it?
- What did the client have to provide that was easy to underestimate?
- What did engineers need after the first handoff?
- Which kind of project would they hire the same team for again?
A detailed answer about an awkward change can be more useful than general praise. It shows how the relationship worked when the original plan stopped fitting.
With the reference’s consent, capture the answer against the relevant worksheet row. If it raises a concern, ask the agency for context before drawing a conclusion. Never turn a private reference into a public testimonial without permission.
What happens when the cheaper proposal looks incomplete?
Return to the hypothetical import problem. Agency A proposes a fixed set of redesigned screens. Agency B includes an investigation of failed imports, sessions with customers, and a review with engineering. B costs more.
The worksheet reveals that these are different scopes. A’s price cannot be treated as the cost of solving the whole import problem. B’s plan is not automatically the better purchase either: customer access and engineering availability still need confirming.
The founder asks A whether the diagnostic work is available and asks B who will run it. If A can support the same brief, the revised proposals become comparable. If the startup already has reliable findings, B’s discovery work may be unnecessary.
This is the useful turning point. The decision becomes specific enough to explain, and either agency can still win on evidence.
When is there enough evidence to choose?
Write a short decision note before committing: the selected agency, the strongest supporting evidence, the remaining uncertainty, and who owns each dependency. Include the condition that would cause the team to reconsider.
If an essential question remains open, resolve that question first. A bounded paid engagement can help when there is work worth doing, but its output and stopping point need agreement. For AI-specific work, the agency evaluation test provides a more focused example.
The finished worksheet should let a colleague who missed the pitches understand the choice. They should be able to see why the agency fits this product, what the startup must contribute, and what still needs checking before work begins.
