In part one of this three part series I recommended that you should thoroughly plan your usability testing activities. I admitted that with unplanned ad hoc testing you definitely will discover usability issues with your system, but you won’t be learning as much as you possibly could with a predetermined methodology and a little planning.
This second part of the the series will focus on a why you should write a test plan and how it may be structured.
Your test plan will guide you through the process of running your usability test. It will set the scene and provide a framework which will define and contain testing activities. It will act as a play by play guidebook for the practitioner and as a valuable deliverable for business stakeholders looking to understand and evaluate the process.
Depending on the type of test you are running your test plan will vary in structure and content but it should follow this general outline. For the sake of consistency I’ll take the point of view that we are testing the sign-up process for the website of a family-friendly theme park.
Why are we testing? Set the scene for your test. You should define what you are testing and why you are testing it. You may have learned from clickstream analytics that a significant percentage of would-be subscribers are dropping off on the second page of sign-up and you want to discover if there is are usability or user experience issues at the root of the problem.
What are we testing, what are objectives of the test? Define what the objectives of the test are. In our case we’ll want to observe users performing the task of signing up for the service. We’ll want to learn what percentage of the participants can successfully complete the task, how many can’t, and what sticky issues they encounter. We’ll want to understand the attitudes of our participants. Did they find the task frustrating, did it take longer than they expected? Finally we’ll want to identify and prioritise areas in our sign-up process that need specific attention from the design team.
Who are we testing? You’ll need to recruit participants for you test, but first you’ll need to define who they are and explain why you’ve chosen to study that particular group. From an exit survey you may have discovered the primary audience for the theme park’s site are parents with young children who are looking for deals on weekend breaks for their family. Through other research you may have learned that this group delivers the greatest level of pounds per user for the organisation and that it’s important as many of them as possible successfully sign-up.
Compensation. Take the time to consider and define what kind of incentive will motivate your audience to participate in your testing. Our parents with children may be keen to help in exchange for a credit note for a high street shop, or a discount on booking accommodation for their trip.
It’s worth noting here that not all audiences will respond to financial incentives. Most legal professionals wouldn’t find a £50 voucher appropriate compensation for an hour of their time. They may be happy to help if they understood that the test would result in improving some software they rely on from day to day.
Screening participants. It’s almost impossible to directly recruit your participants from your target audience. If you’re lucky you may have a database of people that are loosely grouped in the demographic you’re looking for but you’ll still need to screen them for appropriateness.
If you’re inviting potential participants through an email invitation you’ll be able to link those interested to a screener questionnaire, or if your calling people directly you could ask them some questions over the phone. You want to understand if they their behaviour’s would make them valuable test participants. The following screener questions could be included as part of test plan for the theme park project:
Scenarios and tasks. Your test scenarios and tasks are the meat of your testing activities. You’ll be sitting users down in front of your design and observing them performing these tasks. Depending on the complexity of what you’re testing you’ll want to put a good amount for thought into what tasks you want set for your participants.
For the theme part site the scenarios and tasks could be quite straight forward, such as:
Scenario: You are researching holiday options and have learned that Theme Park may be an ideal destination for a family trip. You’ve discovered that there are special rates available for people have registered for the site and you want to sign-up to review these offers.
Task: Proceed through the sign-up pages until you have register a new user account.
Methods and measurement. To make your test worthwhile, and repeatable, describe how you are going to measure and satisfy the objectives you defined earlier. The objectives for the theme park site test were to discover success rates, user satisfaction levels and areas that need specific design attention.
We would set out out testing methodology here, explaining that we would ask the user to think aloud about their decisions and that we would record the screen and the audio. To understand satisfaction we would ask the user to complete a short satisfaction questionnaire after each task with 7 point Likert scale questions.
You’re methods will vary depending on what your testing and what your objectives are, but it’s important to define how you’ll measure your tests and on what scales.
Presenting results. Following on from setting out your methodology and metrics you’ll need to define what the expected output of the tests will be. Perhaps you’ll write a detailed report for business stakeholders, or take the unedited video of the tests to your design team for review.
With the there park test we’ll have gathered information that will allow us to summarise and provide tables that communicate completion times, success / failures rates, satisfaction levels, etc.
With our audio and video we could edit together highlight reels of the most critical issues, and include snippets of users saying things like ‘I’m not sure if this is where I need to click’ and so on.
With your test plan complete you’ll be in great shape and have a solid foundation from which to facilitate your usability testing. In 9 out of 10 cases this upfront planning will give you maximum returns possible from the times invested in testing.
In the final part of this series we’ll put everything together by writing a rock solid test script. The script will allow you or another member of your team to literally ‘play their part’ during the test sessions.
