Usability assessment of Windows Store apps

  • Article

Why it's important to assess the UX of your app

When you improve the design of your app's UX:

  • It's easier for people to use your app.
  • More people will find your app to be valuable.
  • More people will like your app and the features it offers.
  • More people will download and use your app.
  • You will make more revenue from your app.

Assessing your app's design instills confidence that the product you're shipping is the product that you set out to ship, that the user experience is outstanding, and that users will find it useful, usable, and desirable.

Measuring your design's potential success

Start by defining the goals for your app. Goals help you streamline the app-creation process, and they can help you assess your app's success. For more information about setting goals, see Planning Windows Store apps.

There are several stages of evaluation that you can use, depending on the amount of time and resources that you can dedicate to designing your app's UX:

  1. Self evaluation
  2. Cognitive walkthrough
  3. Telemetry, user ratings, comments, and iteration.

Stage 1: Self evaluation

Overview The self evaluation stage is based on the goals that you set previously. The intention of this evaluative method is to ensure that your design is on track with what you intended to do. This step focuses on the overall UX of your app.
Time 15-30 minutes. The time is app-specific and depends on the number of key scenarios in your app.
When You can use this evaluation during the conceptual phase of your app design, and you can use it at any point during development when you want to check progress against your original plans.
Who This evaluation involves one or more of the app designers or developers.
How
  • List the key experiences or tasks that you want the app to provide to customers. For example, in a word puzzle app, a task might be to type a word and submit it.
  • Prioritize this list from most important to least important.
  • Perform the tasks on your prioritized list. As you go through these tasks, reference the goals that you set in the planning phase. Ask yourself questions like:
    • Does your app meet the goals that you originally planned for?
    • If not, what's the status of your app, and what needs to be done to ensure that your app goals are on track?
  • Use the following app evaluation template to document the results from your self evaluation.

Document your experience ratings as follows:

  • Exceeding expectations   Use this rating if your app experience is exceeding your goals.
  • On-track   Use this rating if all of your experiences or tasks are meeting user expectations and there are no concerns or issues.
  • Not meeting user expectations   Use this rating when your app experience has one or more concerns that, if left unaddressed, may dramatically affect the user experience.
  • Concerns with product plan   Use this rating when severe issues exist at the planning level, or when major revisions need to be made to the overall framework of your design.

 

App Evaluation Template: Self Evaluation

The following table is an app evaluation template that contains results from an example self evaluation.

Success metric Goals Status (Date) Comments Issue What's needed to reach goals?
Great At: What is your app great at? What should be the focal point of the visuals? My app is great at providing people with a fun and entertaining experience where they can compete with friends at spelling words. On track None None None
Usable: What activities should users be able to understand, know, or do more successfully because of your app? People should be able to navigate the game, enter words, and submit them. Concerns with product plan None People are not able to navigate between friends when spelling. Need to rethink UI layout.
Useful and Desirable: What about your app do you want your customers to value? What parts of your app did you engineer to delight users or make them love it? Use the table of semantic differentials to assess desire and emotional connection. People should value this app as fun and entertaining. We expect to hear these terms when people describe our app: useful, welcoming, connects me to friends and family. On track Reviewers used words such as "functional" and "welcoming." None None

 

Stage 2: Cognitive Walkthrough

Overview A cognitive walkthrough is an evaluation method in which people perform defined tasks within your app and provide feedback as they do so. This method has a distinct advantage over a self evaluation because the feedback that's provided is from actual users of your app. The most important aspect of this method is that users talk aloud as they complete the key tasks that you've defined.
Time ~30-60 minutes. The time is app-specific and depends on the number of key scenarios in your app.
When You can use this evaluation during the conceptual phase of your app design by using wireframes, and you can use it at any point during development when you want to check progress against your original plans.
Who This evaluation involves one or more users of your app who have been identified as your target audience. Define your target audience by asking the questions like:
  • Who will be using my app?
  • What is the age range of this audience?
  • What key identifiers make this audience unique?
How
  • List the key experiences or tasks that you want the app to provide to customers. For example, in a word puzzle app, a task might be to type a word and submit it.
  • Prioritize this list from most important to least important.
  • Have each user complete the list of prioritized tasks in your app. Remind users to talk aloud as they complete each task. As users evaluate your app, reference the goals that you set in the planning phase. Ask yourself questions like:
    • Does my app meet the goals I originally planned for?
    • Are users able to complete each task?
    • What issues do they have as they complete the tasks?
    • How does their experience completing the tasks compare to the goals I originally set for my app?
    • What is the status of my app and what needs to be done to ensure that my app goals are on track?
  • Use the following app evaluation template to document the results from your cognitive walkthrough.

Document your experience ratings as follows:

  • Exceeding expectations   Use this rating if your app experience is exceeding your goals.
  • On-track   Use this rating if all of your experiences or tasks are meeting user expectations and there are no concerns or issues.
  • Not meeting user expectations   Use this rating when your app experience has one or more concerns that, if left unaddressed, may dramatically affect the user experience.
  • Concerns with product plan   Use this rating when severe issues exist at the planning level, or when major revisions need to be made to the overall framework of your design.

 

App Evaluation Template: Cognitive Walkthrough

The following table is an app evaluation template that contains example results from a cognitive walkthrough.

Success metric Goals Status (Date) Comments Issue What's needed to reach goals?
Great At: What is your app great at? What should be the focal point of the visuals? My app is great at providing people with a fun and entertaining experience where they can compete with friends at spelling words. On track The two users evaluated were clearly able to understand the intentions of the app. None None
Usable: What activities should users be able to understand, know, or do more successfully because of your app? People should be able to navigate the game, enter words, and submit them. Concerns with product plan None Both users had issues with navigation and entering words. Need to rethink UI layout.
Useful and Desirable: What about your app do you want your customers to value? What parts of your app did you engineer to delight users or make them love it? Use the table of semantic differentials to assess desire and emotional connection. People should value this app as fun and entertaining. We expect to hear these terms when people describe our app: useful, welcoming, connects me to friends and family. On track Both users had fun with the app. They used words such as "fun" and "enjoyable to play with friends." None None

 

Stage 3: Telemetry, User Ratings, Comments, and Iteration

Overview The intention of this stage of evaluation is to assess how your app has been received in the marketplace and what you can do to improve the app based on real user feedback.
Time >1 day. The time depends on how much data you collect and what resources you have for analyzing data and creating new iterations. This stage is an iterative process.
When You can use this evaluation after you have released a version of your app and users have had time to find, download, and use it, and submit feedback about it.
Who This evaluation is performed by one or more of the app designers or developers.
How
  • Use your Windows StoreDashboard to analyze your apps in the Store. The Dashboard displays the following info:
    • Adoption: How users find and download your app, how much time they spend using it, and what purchases were made from within your app.
    • Feedback: Ratings and reviews from users.
    • Quality: Exception rates, crash rates, and unresponsive rates.
  • After you gather data about your app, refer to the goals you set for your app during the planning phase. Ask yourself questions like:
    • Does your app meet the goals that you originally planned for?
    • If not, what's the status of your app, and what needs to be done to ensure that your app goals are on track?
  • Use the following app evaluation template to document the results from your analysis.

Document your experience ratings as follows:

  • Exceeding expectations   Use this rating if your app experience is exceeding your goals.
  • On-track   Use this rating if all of your experiences or tasks are meeting user expectations and there are no concerns or issues.
  • Not meeting user expectations   Use this rating when your app experience has one or more concerns that, if left unaddressed, may dramatically affect the user experience.
  • Concerns with product plan   Use this rating when severe issues exist at the planning level, or when major revisions need to be made to the overall framework of your design.

For more information, see Analyzing your apps in the Windows Store.

 

App Evaluation Template: Telemetry, User Ratings, Comments, and Iteration

The following table is an app evaluation template that contains example results from analyzing data after an app was published to the Store.

Success metric Goals Status (Date) Comments Issue What's needed to reach goals?
Great At: What is your app great at? What should be the focal point of the visuals? My app is great at providing people with a fun and entertaining experience where they can compete with friends at spelling words. Not meeting user expectations Adoption is lower than expected in the target market. App listing may not be compelling. Need to revise app's description and screen shots.
Usable: What activities should users be able to understand, know, or do more successfully because of your app? People should be able to navigate the game, enter words, and submit them. On track None None None
Useful and Desirable: What about your app do you want your customers to value? What parts of your app did you engineer to delight users or make them love it? Use the table of semantic differentials to assess desire and emotional connection. People should value this app as fun and entertaining. We expect to hear these terms when people describe our app: useful, welcoming, connects me to friends and family. On track Reviewers used words such as "entertaining." None None

 

Assessing desire and emotional connection

In all stages of UX evaluation, it is important to understand how research participants respond to app experiences. One way to assess responses is by using semantic differentials, or opposite words, like "clear" and "unclear." When you interview your respondents, listen for words that convey desire and emotional connection. Compare the responses of multiple users and look for semantic differentials. Interviewing the participants in your user study can help you understand more about what parts of the product, combined with their personal perspectives, contributed to forming their responses.

Here's a table of semantic differentials. These are potential words that research participants can use to describe their experience.

Words that describe a positive experience Words that describe a negative experience
I'm excited to use this I'm not excited to use this
I'm confident I can achieve all my goals with this I'm uncertain I can achieve all my goals with this
I want to incorporate this in important or common activities I don't want to incorporate this in important or common activities
This makes me feel more satisfied or happier This makes me feel unsatisfied or frustrated
I'm proud to use this I'm not proud to use this
Useful Not Useful
Functional Broken
Fast Slow
Essential Not essential
Welcoming Not welcoming
Compatible Incompatible
Connects me to content and people Disconnects me from content and people
Connects me across devices and environments Doesn't connect me across devices and environments
Works with my individual needs Works against my individual needs
Reflects me or my interests Doesn't reflect me or my interests
Visually appealing Not visually appealing
Premium Inferior
Cohesive Not cohesive
Comfortable Intimidating
Clean Chaotic
Natural Unnatural
Engaging Not engaging

 

Using sets of opposite pairs can help respondents elaborate on their perceptions of an attribute. For example, research has found that the attribute "Connected" means many things, but defining it with the following sets of opposite pairs enables respondents to describe their perception of the word "connected."

  • Ready to go vs. Time consuming
  • Connected vs. Lonely
  • Safe vs. Vulnerable
  • Plugged in vs. Disconnected

When you're considering methods for understanding how desirable or your app's experience is, focus on the words that you hope new and experienced users would use during a conversation with a trusted friend. Prioritize the most important items. The entire experience should elicit positive emotions, but sometimes putting more energy toward one part of the experience means that the emotional response in another area is reduced. It's important to know what to expect and to listen for the things that are most important to you.

Planning Windows Store apps

 

 

Build date: 9/20/2013