How can a non-designer judge a design agency’s work?
Three portfolios all look good, and taste cannot separate them. Checks a founder who is not a designer can run on a case study and a first Figma file, with a scoring sheet and five walkthrough questions.
A founder has three agency portfolios open in three tabs. Each case study has clean screens and a one-line result. The founder is not a designer and cannot say which work is better. Taste does not help, because all three look good.
Taste is also the wrong tool. Final screens show the result of decisions, not the decisions. You can inspect the decisions: is the problem stated, are the flows complete, is the file consistent, accessible and buildable, and can the team explain its choices?
This guide applies those checks to a case study and to the first deliverable you receive (flows, wireframes or a Figma file). Portfolio relevance, proposals and reference calls are in our guide to evaluating a design agency when every pitch looks good. Humbleteam publishes this guide and is a design agency, so run the sheet below on our work too.
What should a case study show besides final screens?
Nielsen Norman Group (NN/g) surveyed 204 UX professionals who hire designers about what they look for in a portfolio. According to NN/g’s portfolio guide, they want the problem, the person’s role, the constraints, the timeline and how research shaped the design, not only the finished product. The guide is written for designers, but an agency case study answers to the same questions. NN/g’s outline adds the concepts the team did not pursue and the effect on users and the business.
Cover the screens with your hand and read what is left. A strong case names the problem in a sentence, names what limited the team, shows one idea it dropped, and shows how it checked the result.
On evidence, “tested, changed, tested again” beats “users loved it.” Nielsen’s often-cited estimate, published in 2000, is that 5 participants find about 85% of a design’s usability problems, so he recommends small tests with fixes in between. A small test cannot prove a business result. A case that claims one should say what was measured and when.
What should you check in the first deliverable?
Are the flows complete?
A flow with only the happy path looks finished and is not. Pick the flow that matters most to your business and ask to see what the user sees when there is nothing yet, while content loads, when something fails, and when they lack permission.
NN/g explains why empty states matter: a blank screen leaves users unsure whether the system is working, whether an error occurred or whether they set a filter wrong. For errors, NN/g’s usability heuristics ask for plain language, a precise description of the problem and a suggested fix. “Error 4012” fails that test. The permissions state is our addition, not a published rule: if your product has roles, each role that sees a screen differently needs its own version.
Is it consistent?
NN/g’s Design Systems 101 says reusable components reduce the risk of unintended inconsistency. Figma supports this. An instance stays linked to its main component, variants group states such as default, hover, pressed and disabled, and variables store values such as spacing.
You can test this without design skills. Click a few buttons on different screens. If they belong to one named component, the file was built from a system. If each is a loose rectangle with its own corner radius, every change will be repeated by hand. Read the labels too: if one screen says “Remove” and another says “Delete” for the same action, ask which is right.
Is it accessible at a basic level?
A design file cannot prove accessibility, which is judged on the built product. It can show whether the basics were considered. Ask for:
- Text contrast of at least 4.5:1, or 3:1 for large text of 18 pt or 14 pt bold (WCAG 1.4.3), and 3:1 for the borders and icons that identify a control (1.4.11).
- Targets of at least 24 by 24 CSS pixels in WCAG 2.2 (2.5.8). Apple’s guidance for buttons is a 44 by 44 pt hit region. Android’s guidance suggests 48 by 48 dp.
- A keyboard focus state drawn in the file, not only hover (WCAG 2.4.7).
Ask for the contrast ratios of the color pairs used. Any free contrast checker can confirm them in a minute.
Can engineers build it?
Figma lets a designer mark a frame as Ready for dev and flags it as Changed if it is edited afterward. Annotations carry notes in preset categories: development, interaction, accessibility and content. Look for them, and for consistently named components with states as variant properties.
One practical test, which is our advice and not a standard: give the file to your lead engineer for 30 minutes and ask for a written list of questions. A short list for a key flow is a good sign.
A scoring sheet for one case study or one deliverable
Mark each row Strong, Partly, Weak or Cannot tell. Do not add the marks up, because a single total lets a polished surface hide a missing flow. “Cannot tell” is not a fail. It is a question for the walkthrough.
| Check | Strong looks like | Weak looks like |
|---|---|---|
| Problem and constraints (case) | One sentence says who was stuck, where, and what limited the team | A mood word such as “modern” and no user or constraint |
| Decisions and trade-offs (case) | A rejected option, with the reason | Only the final design, as if no other path existed |
| Evidence (both) | States what was tested, what changed, what was measured and when | “Users loved it,” or a result with no method |
| Complete flows (file) | Empty, loading, error and no-access states drawn, with plain error text | Happy path only, or an error shown as a code |
| Consistency (file) | Same named component, spacing from variables, one label per action | Loose shapes, mixed radii, two labels for one action |
| Accessibility basics (file) | Ratios listed and meeting 4.5:1 and 3:1, platform minimum targets, focus drawn | No ratios, small targets, hover only |
| Buildable (file) | States as variants, Ready for dev used, notes on interactions and content | Layers named “Frame 214,” missing states, no notes |
| Reasoning (walkthrough) | Names the user and the constraint, and what was traded away | Defends taste (“it feels cleaner”), cannot find the source of a decision |
How do you ask an agency to walk you through a decision?
Write the objective first. NN/g defines a design critique as analyzing a design and giving feedback on whether it meets its objectives, and says that without agreed objectives, feedback is subjective. Its expert review guidance adds that a problem should never be called a problem because the reviewer dislikes something. So before the call, write one sentence, for example: “A manager can approve a payment request without leaving the list, using our existing components.” Then choose one screen and ask:
- Which user problem does this screen solve, and how did you learn about it?
- What did you try first, and why did you drop it?
- What does the user see here if data is missing or the request fails?
- Which constraint cost you the most, and what did you give up for it?
- What would you test next, and what result would change your mind?
A team that names the user, the constraint and the trade-off gives you something to check. A team that answers “it feels more modern” gives you taste back.
An illustrative example: two files for the same brief
This scenario is invented. It uses no real agency and no real numbers.
A founder of a B2B invoicing product asks two finalists for the first flow of an approval screen, with the objective above. The founder fills in the sheet from each case study and first file.
| Check | Agency A | Agency B |
|---|---|---|
| Problem and constraints | Partly: opens with a mood, names no constraint | Strong: approvers drop requests with missing fields; existing component library |
| Decisions and trade-offs | Weak: final screens only | Strong: a dropped modal concept, with the reason |
| Evidence | Weak: “much improved,” no method | Partly: one 5-person test and one fix, no retest |
| Complete flows | Weak: 14 screens, all happy path | Strong: 11 screens including empty, error and no-access |
| Consistency | Weak: three button styles | Strong: one button component |
| Accessibility basics | Cannot tell: no ratios given | Partly: ratios listed, no focus state |
| Buildable | Cannot tell: unnamed layers, no notes | Strong: states as variants, notes on error handling |
Agency B is ahead, but the sheet does not say “hire B.” It says what to ask next. Agency A gets questions about its contrast ratios, its error states and the options it dropped. If A answers with specifics and a revised file, A stays in. B gets one: will it retest the fix? “A feels weaker” has become a list of things each team can fix or fail to fix.
If two agencies are still close, a paid trial project is the next step, and the same sheet works on what each delivers. Before your next call, pick one case study and one flow, write the objective in a sentence and send the five questions in advance. Fill in the sheet during the call, not after it.
