Steam is an online video game platform created by Valve that offers access to more than 30,000 games, from AAA franchises to independent games.

Gamers can log in to the website to conveniently purchase and play games online, a better alternative to buying physical copies of the games and manually downloading it on the computer.

1. As one of the users to Steam, I always feel frustrated every time I try to navigate in Steam, finding the functionalities I need.

Some friends around me also had similar circumstances ---- the interfaces are cluttered, crowded, and not enough white space. It largely decreases the purchase experience within the product. As a team of 4, we want to discover the current user pain points through usability testing.

‍

2. Increased popularity during the pandemic

People were forced to stay at home during the pandemic, this causes a huge increase in the number of customers buying games. We believe it will be interesting to continue to observe the impact of the pandemic on people’s gaming habits.

‍

‍

The Goal of the Test

"How easy it is to navigate between different functionalities in Steam?"

In addition, we also want to test the learning curve of users and whether Steam can be learned quickly.

‍

‍

Define Target Audience

We conducted online research to gather information about target audience of Steam

1.Novice Users

Novice users are people who are new to Steam for various reasons.  For this user group, an intuitive and straightforward onboarding process would be essential for them to start playing instantly. They are more likely to choose free/cheaper, simpler, and collaborative games. 

‍

2. Casual Users

This user group takes up to 56% of the PC gaming audience. They prefer more relaxing and gentle games such as Minecraft and Cities Skylines. This user group likes to view the game details page and the reviews beforehand so that they can make decisions.

‍

‍

Design the tasks 📝

We asked the users to imagine that they are on a day off, and one of their friends invited them to play a game with him/her on Steam. The test begins with an assumption that they have downloaded the desktop application and installed it on their computer. The situation was set so that this is the first time they are opening the steam application.

In total, 8 tasks were designed:

• Task 1: Could you please walk me through how to create a new account? And Login in to your account. You don't need to click continue after you fill out the information. 
• Task 2: Now please use this account to login. Email: xxx  Password: xxxxxxx
• Task 3: Please freely explore the interface of Steam. What elements of the interface attract your attention the most? Could you tell me what your first impression of the interface is?
• Task 4: Find a game called Cities Skyline and review the description. 
• Task 5: You find that there are many bundles of the game. What is your next step? Please compare these bundles and find one that you think is most suitable, then add to cart and purchase it. 
• Task 6: Find the community group of Cities Skyline and follow the group. Could you tell me your impression of the community page?
• Task 7: Imagine you have downloaded the game, you would like to add your friend and play with him/her together. The account is xxxxxxxx. Could you show me how would you do it?
• Task 8: Now, you want to find a game in the casual category which is free and can be played by MAC OS. How would you do it?

‍

‍

How to Set up the Test? 💻

Testing facility and equipment:

• The test will be conducted in-person. A laptop (either Mac or PC) is required, and a mouse will be prepared according to the preference of participants.
• Steam will be installed prior to the test session begins, and we will use Zoom to capture participants’ interactions within Steam.
• During the test, notetaker and observer will each have a laptop to record their observations and findings, as well as the answers of post-session questionnaires from participants.

‍

Duration of test sessions: 20-25 minutes for each participants

‍

Data to be collected:

• Interactions and steps participant takes to achieve each task
• Errors participant makes during an task
• Useful information and questions as participants apply the Think Aloud Protocol during the test.
• Questioning their behaviours from an observer

‍

Roles of each team member:

We arranged roles to each member and in order to not miss information, observer will also take notes along with the notetakers.

• Moderator: Yanan, Guru (take turns)
• Notetaker: Churan, Guru, Jingyu (take turns)
• Observer: Jingyu, Guru (take turns)
• Facilitator: Churan

‍

Participant Profiles

All 6 participants are graduate students within the same course. According to the background questions they answered, here are their background profiles:

‍

During the test, the session was recorded using Zoom to capture their interaction with Steam. Demographic and psychographic information were collected using background questions. Then, the moderator guided the participant through a set of tasks by giving them corresponding scenarios.

There were 2 people taking notes so the moderator can be 100% focused on the relationship with the participants. In the middle and at end of the session, we asked participants what is the task(s) they felt mot difficult and easiest to perform. Lastly, we asked any follow-up question and let them fill out product reaction form

‍

Moderator Script

To keep the methodology consistent and determine how long a session takes, we created the moderator script to guide us through the session.

It consists of 5 parts: introduction, background questions, tasks script, post-session questionnaire, and product reaction form.

‍

Participant Performance

We also noted down some of the noticeable participant performances during the test:

• Participants finished most of the tasks successfully and quickly.
• Participants could easily use the search bar to find the specific game on the store and community page, but they found it hard to distinguish the differences between each bundle.
• A few participants were only able to find the community group of Cities Skylines but failed to find the "follow" button
• Participants scrolled up and down to find the username search bar to add a friend to complete the task, and they were confused about the difference between "Friend Code" and "username".
• Most Challenging Task: Although the participants could easily select a casual or free to play game from the navigation bar, they noticed that multiple filters cannot be applied simultaneously, so they had to manually select the game that supports the Mac platform.

‍

Gather Findings

We recorded each participant's answers, behaviours, and our observations and findings (positive & negative) in Google doc.

‍

‍

Affinity Map

Then, we synthesized positive and negative findings from each participant using affinity map to discover important insights.

‍

‍

Rate Severity

We rated each finding based on their severity from 1-4 to help determine which problems should be fixed first:

1: Cosmetic problem ❗️
2: Minor usability problem ❗️❗️
3: Major usability problem ❗️❗️❗️
4: Usability catastrophe ❗️❗️❗️❗️

According to Nielsen's 10 heuristics, we wrote down which heuristic has been violated in each finding.

For each finding, we proposed some recommendations that could be possibly used to fix that problem.

‍

In total, we found 1 usability catastrophe, 5 major usability problems, 6 minor usability problems, and 5 cosmetic problems.

‍

‍

Participant Impression

According to the product reaction form results, we found following insights:

1. 4 out of 6 participants agree that there is too much inconsistency in Steam, suggests inconsistency is one of the problems Steam has.
2. The technical threshold of Steam is not high and most participants (5 out of 6) suggest they are able to get started without technical help.
3. There are 5 out of 6 participants indicate that they imagine most people would learn to use Steam very quickly.

‍The final report is consisted of following parts:

1. Executive Summary
2. Introduction
3. Test Procedure
4. Participants
5. Findings and Recommendations (includes screenshots and user quotes)
6. Participants' Impressions (From product reaction form & debriefing interview)
7. Conclusion

‍

In final report, findings regarding "Interaction" and "Visual" of Steam were added. For example, we observed participants instantly learned and relied on hover to understand game reviews, menus, game information, or various bundles. One participant mentioned, "I like the dark theme, it is obvious for a gaming platform and it has some twitch vibe”.

1. Integrate Findings into a Redesign

Now I've known the usability issues of Steam, I'd start think about how I can redesign the interface to solve these issues, especially the inconsistent layout.

2. Expand the diversity of participants

Since this is a course project, we were only able to find participants within the same class (similar ages, similar tech level etc.). If I had time, I would conduct more usability tests with target users who are not within this age range and having different backgrounds, as this can help me find more edge cases and discover unexpected issues.

Always run a pilot test !!! We didn't have a thorough enough pilot test before the formal started, and the participant was one of the team members who was already familiar with the tasks. These caused several issues during the formal usability test sessions and wasted some time. So it’s best to get a pilot test participant who closely fits the profile of the actual participants I’ll be using.

Becoming a good moderator needs practices. It was hard to always be conscious about some of the affirmations (e.g., ‘Mm hm’, ‘Okay’, ‘Uh-huh’) and give prompts and assists when needed. It would be better if we can encourage a hesitating test participant to let them know we are engaged and listening.

Preparation is the key! I noticed when the moderator read a task to the participant, some participants could not remember what the task is. The more effective strategy would be to provide them with a copy of the task, either on paper or on another screen.

‍