.
Less is more (effective). Here's why big, expensive usability tests might not be a good use of resources. Can the best results come from testing just 5 users?
PropertyLab is in full swing and feedback on our innovation programme has been nothing short of incredible. Over the next few weeks, PropertyLab will tackle some of the biggest issues landlords face in keeping their properties safe by focussing on the key aspects of the regulatory compliance process in the residential sector - specifically fire and gas safety.
At Plentific, we understand the importance of effective research - it's the foundation on which our product design decisions rest. But there's a lot to getting research right. And sometimes, it isn't as clear when you've gathered all pertinent insights and it might be time to stop.
So this update features the concept of usability testing (more on this below) - the goal being to showcase why traditional mindsets on the number of users in a product design study might not be as effective as keeping these studies limited to small numbers.
How many users does it take?
Many feel that usability (a product design exercise that determines user-friendliness) is very costly and complex. In the housing context, teams need to know how many tenants or property managers it will take to confidently push a new product in the right direction - schedules, budgets and the project's success can depend on it.
Which is why a traditional mindset might limit such an exercise to that rare behemoth of a web project. You know, the one with a big budget and all the time in the world. That shouldn't be the case.
How about just 5?
In 1993, Jakob Nielsen, an expert on usability, described a key finding from his studies.
Jakob holds a Ph.D. in human–computer interaction and, amongst other accolades, has coined Nielsen's Law, which states that network connection speeds for high-end home users would increase 50% per year, or double every 21 months.
Part of what inspires our PropertyLab research (and this update you're reading) is a movement Nielsen founded on "discount usability engineering" - what he likes to call fast and cheap improvements of user interfaces. You can access his findings on usability in full here.
Jakob's analysis showed that with a sample of only 5 users, one can uncover 75–80 % of design usability issues.
Anything more would be a potential waste of resources.
Let's see why.
You've learned enough.
Just with a single test user, insights gathered shoot up to 33% of what you need to know about usability. There's some overlap with the second test user, so not as much new insight. With the third, you'll already see visibile repetition. Only small amounts of new data are generated with any additional person in the mix. At 5 people in, you won't find anything new.
Source: The Right Number of User Interviews, M. Seaman. Medium.
Amongst a homogenous cohort, usability testing can feel repetitive after just a few interviews. We've confirmed this over the course of the interviews we've conducted as part of our programme's research phase. The insights on access issues, disconnected systems, visibility, ownership, even data, start to sound familiar.
Magical or controversial?
As you might've guessed, such a bold outlook has some skepticism associated with it. 'Treat it case by case' is something quantitaitve experts love to emphasise.
We think it works well for us, the sector and our regulatory compliance focus with our first cohort. One of the main advantages is the ability to dig deeper.
Where others might conduct a usability exercise with 20, we'd prefer to do 4 studies with 5 users each. That means every subsequent study let's us further assess issues like data architecture, flow and structure - without getting bogged down in early stage usability issues (which are common and not as insightful).
The man himself.
Jakob explains how foundational usability testing is best with fewer participants - so that you have time and budget to test more design iterations of the user interface.
Why this is important to PropertyLab.
We put a lot of emphasis on getting phase I (the qualitiative research) right - so we can nail our efforts to get regulatory compliance right. We believe every new insight is important. Which is why we sift carefully through all the ideas, experiences, real-life cases our cohort shares with us through open ended interviews - which are then broken down into topics and subtopics ahead of phase II (discovering and prioritising). This helps us get as close as we can to a complete understanding of every expertise on the programme. We interview until the learnings taper or when we start to feel a collective sense of having learned as much as we can.
Download now
