Hosting my first Usability Test at a Startup

Photo by David Travis on Unsplash

I was sitting on my desk staring at the screen waiting for my first participant of my first usability test. The company I used to work at was involved with educating others. They educate the learners about data science and AI. Our platform hosts various courses and bootcamps. The best part is they would teach you, for free. The goal is to democratize the education of data science and AI. This was a noble cause and I liked working here because of it. But noble cause alone is not always enough to achieve your goals.

I kept waiting for this participant. Five minutes had passed since the meeting had started and nobody had joined. I decided to write a reminder email to the participant. This will be the second one. I had already sent them one, first thing in the morning. Another 10 minutes passed and I decided to label this interview ‘unsuccessful’ and leave the meeting.

As I wrote an entry in the Google sheet about why the meet was unsuccessful, I thought if the process I had followed lacked something. Something crucial. Maybe that’s why I keep getting into meets where the participants doesn’t show up. Or maybe it is that the participants are not seeing any benefit for them in this whole activity. Or maybe it is something else entirely. I was in doubt and confused. There was no senior UX designer at my company to guide me. This was an early stage startup. We barely had 20 people in the whole company. Besides me, there was only one other designer here who is equally skilled and equally new as myself.

All of this started 3 weeks ago. My company had decided to launch a couple of new features which will enhance the overall experience of any learner. But this time, we wanted to test these features with real learners. We also wanted to know what they think of the existing system. This only meant that we needed to conduct a usability test of the platform. And thus, the task was given to us. ‘Us’ as in the two designers at this company — myself and Sankalp.

To be honest, we were quite aloof about hosting a usability test. We had never done this before but in our experience at a startup, this was quite normal. Many times before, we had learnt a skill in one day and applied it in field the next. So with same enthusiasm, we started researching and educating ourselves about usability tests. We scoured the internet and discovered various articles and tutorials. All of them helped but in pieces. There was no one place where we could see the procedure of hosting a usability test in an end-to-end manner. In the end, we filtered the steps that were suitable for us and made a plan of action. This is what it looked like:

Stages of Usability Test | Drawn by yours truly

Usually a usability test is done after design and before development in the product cycle. But in my case I was testing and getting feedback of an already existing system. So I didn’t need to worry about the prototype. I was going to use the actual platform for my tests.

I had planned each stage of the test in detail. And mapped the plan in a google sheet document to make it actionable and easy to follow through. I decided to review each of these sheets to see if I can locate any error in our ways.

Finding the participants

During my research, I’d found out that you can host only 5 tests and it could tell you 85% of the usability problems with your product. But from where do these 5 people come? Should I randomly select 5 people? Can they be someone I know personally (this will be the fastest way to get participants)? Can we do this with 5 people from our own company (they know the way the platform works so they must have some idea about navigating through the platform)? Or should I just ask the community and select the first 5 responses (these are actual learners who seem to be eager for such an activity)?

Luckily, I did not follow any of these initial ideas. You only need 5 tests but in these 5 tests you need 5 participants who are users of our and similar products and cared enough to see how these products affect their lives. Only then they could provide appropriate feedback.

To find these 5 lucky participants, I had created a survey. It included questions that would reveal to me their online learning habits. How often do they use online learning systems? How many platforms have they used so far? What do they think of online learning in comparison with offline learning? What are their expectations from online learning platforms? And so on.

And to get maximum responses from our community, I partnered with the our developers and community manager. We hosted a notification banner on our website linked to the survey. We emailed participants of the recent bootcamps about this activity. We put a message in our Discord server and asked the members to fill the form. It was necessary that this initial survey remained short and sweet. Taking no more than two minutes to complete. Because even though participating in a usability test and giving feedback directly to the designers sounds fun, nobody would fill a lengthy survey knowingly that they might not be selected for the test. And we had ensured that. (Our community manager took one and a half minutes to complete the survey.)

They survey received 78 responses over a week. From these 78 responses I had to shortlist 5 individuals. But for worst case scenarios, I had decided to select 15 individuals divided into 3 buckets. (Why thrice the needed amount you say. Because I knew that filling the survey took less than two minutes, but participating in a usability test can take almost an hour. And when the push comes to shove, not everyone is willing to participate.)

Now, all products have two types of users — power and regular. A power user utilizes most of the features in the product. For them, anything that improves their workflow even by little amounts is good. A regular user will not use all the features but focus mostly on the core usage of the product. Too many enhancements can actually slow them down as they spend too much time learning how to use them. This can harm their experience. However, both type of users can use the product easily if the product has high usability. I ensured that each of my three buckets had at least two power learners and two regular learners.

Bucket list of all potential participants

Communication

After a participant was selected, I had to tell them. I had to inform them about what would happen, when it would happen and what are we expecting from them. Chances are that they probably forgot about this activity while I was busy nitpicking whom to invite in the next stage.

Test invite

The first email I would send to a shortlisted participant. In this, I would explain why are they receiving this email and what will happen next. I ensured that they understand that we will test the platform and not them so they don’t need to prepare anything except joining the meet from a setup which can support screen sharing. As for the meet timing, I used Calendly to manage meetings. This helped in keeping the timings flexible and suitable for both parties.

Test reminder

I would send this at least 6 hours before the meet. It was just a short summary of the invite reminding the participant that they have meeting scheduled with me.

Thank you note

Participating in a usability test can be cumbersome and time consuming. Sending them a thank you note is the least I could do.

Communications

Alas, all I had sent till now were invites and reminders. Due to no shows in all cases I was yet to send any thank you notes :(

Scenarios

Planning is important in achieving a goal. I could not have just invited the participant and given him the platform to play with like how you give a toy to a toddler. They might just end up clicking stuff randomly. I needed some restrictions in the way they would interact with the platform. Some goal that they would try to achieve. But this should not limit them to a certain way of doing things. What I needed were scenarios. Situations that would simulate a brief moment in a real learner’s experience. Situation in which they try to achieve a certain goal but there should be multiple paths to achieve the said goal.

It is important to note that this train of thought is very specific to my case here. I had not read about it in any tutorial. My company wanted feedback on the entire learning journey, not just one or two features. Therefore, I needed to fit in the big picture within a short meet.

I worked with my design buddy Sankalp and we drafted 4 scenarios. They were something along the lines of giving our tester a background of a learner, say somebody who is in college looking to increase their knowledge about data science, who heard about us from their friends and ended up on our homepage. Now the tester is required to wander, find a course suitable for a beginner learner, register and practice the chosen course. Specific yet achievable through many paths.

Scenarios

Given we had 4 general scenarios, each of these had some tasks common among them. These were something very specific such as being able to register for a course, navigate the content of a course, etc. These specific tasks were there to help me out when I would evaluate the test. I could have explained this to you now if only I had completed a test successfully :((

Script

I would like to stress (again) that planning is important. I was going to have an interaction with a person about our platform for maybe as long as an hour. I needed a script to follow so that we could have a moving conversation. It would also help me in staying on track and getting the participant (or myself) back to the topic if we started wandering off on a tangent.

Set Up and Welcome

For starters, I wrote an introduction of myself and my teammate. I had explained what the session was about and what were we going to do. I had to nudge them to be honest with their feedback and not care about my feelings. I also sought their permission to record this interview for future references. I had to ensure that their privacy is respected and only the people involved in this project will have access to the recordings. It was important to get a green light on recording the session because it would help me in revisiting the conversation later in case I had to.

About Yourself

This was the icebreaker. I had to get the participant in a comfortable zone so they could speak without being asked to. The best way to do this was to ask them about themselves. I had written down questions asking them about their experience with online learning or other learning platforms, their hobbies, their professional goals, etc. I had also written some questions that I had removed from the survey earlier, such as any hurdles they might have faced in online learning.

Set up screen sharing

Screen sharing is the back bone of remote usability tests. But not everyone is used to sharing their screens on all meeting platforms. I was using Google Meet, so I had written instructions on how to share one’s screen on GMeet. I had already mentioned about this in the invite and reminder email, so I was hoping the participant to not act surprised when I ask them to do this.

Perform Tasks

I had noted each of the 4 scenarios I told you about earlier and the common tasks among them. I had to ensure that the participant understands the scenario clearly so I had planned to read out the scenario allotted to them and also text it to them in the meet. After that, they would steer the test themselves. All I had to do was sit back, observe and keep nudging them to think aloud.

Follow Up

I had not written down much here because this section would depend on the interview itself. If I had something interesting mentioned by the participant, but did not want to break their flow, I would ask them more about it here. And in the end, I would ask them to complete the SUS form.

Wrap Up

This was the bye-bye. I thank them for giving me their time and feedback. I had written some questions about the usability test in general, for example, if they had liked it, would they be interested in doing it again, etc.

Script

Observations

The end goal of a usability test is to come up with actionable problem statements about your product which will help in improving the usability. To do this we need real users and real observations. As a learner would be enacting the scenario allotted to them, I had to note down any and all observations. I had divided the feedback column into positives and negatives for any topic. For example, if the topic is registering for a course, and one participant finds that they were able to locate the register button easily but got confused with a field in the registration form, then their first point would be positive and the second one a negative.

Similarly, I had made a list of various topics linked to the tasks which were common in every scenario. And each topic had a positive and negative feedback column. This practice would help me in sorting the feedback early on. It would reduce efforts in later stages.

Task Observations

I had another observations sheet which was to be filled before this one. It corresponded to the questions I would ask in the About Yourself section. It’s always a good practice to keep a record of all answers from your participant because it helps you in understanding their sentiment and feedback later.

About Observations

I rolled my eyes and scoffed in the air. I could not locate any error in the way I had carried out the whole process till now. Getting the participant on the basis of mutual interest is not entirely in my control. I reminded myself of this and closed the lid of my laptop.

The first bucket was completely unsuccessful with 4 people not responding and 1 person not showing up after scheduling the meet. I had already send the invite to second bucket and was happy to see that 3 of them scheduled interviews. Two of them were going to happen tomorrow. So for now, I concluded that my preparations were enough. I crossed my fingers and hoped to conduct successful tests tomorrow.

Performing the test

The next morning I joined the test meet link scheduled by the participant. I was joined shortly after by Sankalp, my teammate. We chatted for a while and decided our roles for the test. I was to host and conduct the majority of it while Sankalp will try to make notes and observations. We were discussing something unrelated when my speakers pinged in the same way when somebody joins a meet. Our participant was here. Finally.

I greeted her and introduced myself. Sankalp did the same. Our participant introduced herself. She was a college student studying Electronics and Communications engineering. I began following the script and informed her about the session and what would happen. She was courteous to allow us to record the meeting. Then I asked her questions about herself to break the ice. This worked as she warmed up and started getting more descriptive with her answers. She even started mentioning her expectations from online learning platforms. Sometime later, I decided to shift gears and asked her to share the screen. She was able to do it easily and pointed out that if we had not asked her to join from a suitable setup, she would have joined from her mobile. I gave her one of the scenarios. She was able to understand it and started exploring the platform. She completed all the tasks linked to her scenario. There were times I had to remind her that we need to hear her opinions as they come, so she needs to say them out loud. A little over half an hour had passed till now. After completing the tasks, I asked her a few questions as per the follow up section. I told her about the SUS form which she quickly completed. At the end, we discussed how this activity was and thanked her for her time. I was really happy to know that she enjoyed the whole session and look forward to participating in more of them.

And just like that, I had hosted my first usability test. (Drinks all around! 🍻)

After composing myself (ahem), Sankalp and I readied ourselves for the next round… I mean the next test. We were hopeful that this one will go great as well. Soon, we were joined by our participant from Nigeria.

I carried the test in similar fashion as before. Throughout the week Sankalp and I had conducted 5 successful interviews. We kept exchanging our roles between ourselves. Two of the five participants came from the third bucket. Keeping a safety net of 10 extra people did pay off in the end.

After completing the aimed number of usability tests, the next part was to analyze all the insights and generate actionable problem statements. Quite a few times we had to revisit the meet recordings because we were unable to keep up with pace of the test or because we focused too much in conversing with the participant. It was supposed to be a two-way conversation after all :P

This was really fun :)

Analysis

You might think that analyzing the information which you have gathered after so much work should be easy. It might even be for some usability tests, but in our case I had made the participants explore the entire platform, click at multiple places and make various decisions. Everyone had something to say. Somebody requested a new feature, somebody put forward their personal opinion, somebody commented on something totally unrelated. But all of this was feedback. I needed to sort it out to understand it better. Splitting the feedback into positives and negatives gave me an edge here. I can focus on the negatives. But from my research I was aware that to analyze the findings we need a plan of action. The first step was to calculate the SUS of your system.

Took me a few attempts but I finally drew him right :)

System Usability Scale (SUS) is a metric used in the design industry for the last 36 years. It’s a fairly cheap and quick method to determine the usability of a system with help of statistical data. SUS is measured with the help of survey. The survey contains 10 questions based on Likert scale. Each question is a statement about the system. Participants will rank each question from 1 to 5 based on how much they agree with the statement they are reading. 5 means they agree completely, 1 means they disagree vehemently.

After I had gathered the ratings of all participants, I calculated the SUS score with the required formulas. I’m skipping the instructions for the calculations for now but you will hear them at the end. The industry standard is 68. After calculating, and recalculating because I made an elementary maths mistake in the first attempt (“What a noob” is what Sankalp must have said when I told him about my mistake…), I discovered that we achieved a score of 78. Not very high but it certainly passed the industry standard. This meant our platform was usable enough for an average person.

Let’s get back to the feedback of the participants. It was time to sort them. Since I only focused on the negative feedback, one can say I was only looking at the ‘issues’. And to solve these issues, I needed to understand their importance with respect to each other. For this, I needed to assign each issue a priority.

I gave each issue and ID. It was simply a serial number. After that, I gave them a description and location. Description explained what exactly is the problem and location pointed out where the issue is occurring on the platform. After this, I added columns labeled P1 to P5, indicating the occurrence of a particular issue for each participant. For example, if the first and fourth participant faced the issue 7, then P1 & P4 will have a ‘1’ for issue 7 and P2, P3 & P5 will have a ‘0’. This gave me the frequency of each issue. Taking the example stated above, the frequency would be 40% or 0.4 (2/5 * 100 = 40%).

But frequency alone cannot justify the priority order. What if there is an issue in a task which is crucial for the business (task criticality). Or what if there’s an issue which impacts the task so much that the learner cannot achieve their goal (issue impact).

I added a column for task criticality. I rated each issue based on how important was the task to which the issue was related to. The ratings were something like this:

• 1 — Low
• 2 — Medium
• 3 — High
• 5 — Critical

I added another column for issue impact. I again rated each issue, this time based on how much did they affect the task to which the issue was related to. The ratings were like this:

• 1 — Suggestion
• 2 — Minor
• 3 — Major
• 5 — Blocker

I calculated the priority of each issue as the product of its frequency, task criticality and impact. Sorting by descending order gave me the sequence in which I needed to work on these issues individually.

Sorted list of issues

Conclusion & Next Steps

Along with my teammate, I had successfully conducted the first usability test of our company. The issues we discovered will be discussed and picked accordingly by design and development team in the coming future.

In this past month, I had read many articles on usability testing, learnt many different things and practiced most of them in a real-world scenario. I remember that all I wished for was if I could find everything I’m looking for at one place. Hence, after completing my report on the test I decided to to tell the story of how I hosted a usability test, notably at a startup. And also, my first usability test. And here we are.

If I were to point out the biggest hurdle I faced, it would certainly be the lack of rewards for our participants. I had spent ample time in coming up with a substitute for monetary rewards. In the end, I had to trust the community members in helping us out. Lack of resources is a scenario which can be often seen at early stage startups. And I believe it might have played a role in the early unsuccessful tests.

If you stuck with me till here, thank you very much. Let this article act as a guide for you so that you can host a usability test by your own. In the next section you will find some helpful resources which I had explored in my learning journey. Use them. Use them and host that usability test.

And don’t forget to tell me all about it ;)

Resources

Getting Started

• The Quick & Simple Guide to Starting User-Testing for Startups (designforfounders.com)
• Why user testing matters for early-stage startups (slowlettuce.io)
• Usability Tests to Take Your Startup to the Next Level (spdload.com)
• How to Conduct Usability Testing (startup.info)

Finding participants

• A Guide to Finding Free and Paid Participants for Usability Testing | Marvel Blog — Marvel Blog (marvelapp.com)
• Recruiting Test Participants for Usability Studies (nngroup.com)

Communications

• Usability Testing Recruitment Email (2022) | PlaybookUX

System Usability Scale (SUS)

• How To Use The System Usability Scale (SUS) To Evaluate The Usability Of Your Website — Usability Geek
• The System Usability Scale & How it’s Used in UX | Adobe XD Ideas

Analysis

• How to Analyze Usability Test Results | Toptal
• How to Analyze and Report Usability Test Results | Maze
• Measuring user experience with usability metrics (maze.co)

🎁 Here’s a gift for you…

I made a template of the sheet document that I used throughout the whole test. Trust me, documentation is one of the most important aspects to a successful usability test. Make a copy and have fun!

Usability Testing (Template) — Google Sheets

Fun fact: You can click and hold the 👏 icon to give multiple claps on Medium. Go ahead and try it out.