Fusebox

Often times we’re asked to perform system usability studies to derisk the launch of a new site. If it’s late in the process (not atypical) one of the first things we’ll do is come up with a high level subjective view of usability. We use our in-house team as well as our network of usability consultants to come up with a simple assessment.

Our assessment uses the popular system usability scale (SUS) framework to come up with an industry standard score that evaluates the system as a whole. SUS comprises a high level questionairre that by design is quite general. We want to identify if the system in question is inherantly problematic or not. If it is problematic, we’ll dig deeper, if not, we’re ready to move on.

The questionairre asks a user to respond from a list of statements using a five-point Likert scale that ranges from Strong Disagreement to Strong Agreement.

Here’s a sample per question response that I’m sure you are all familiar with:

Here’s the standard questionairre. You can also find it over at usability.gov.

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

Scoring is simple and normalized to a 100 point scale. There is a possible score per question of either 0, 1, 2, 3, or 4. In order to ensure no bias towards any of the statements with respect to agreement or disagreement, questions 1, 3, 5, 7, and 9 give more points to Strong Agreement (ie Strong Agreement = 4 points), while questions 2, 4, 6, 8, and 10 give more points to Strong Disagreement (ie Strong Disagreement = 4 points).

In order to base this on a 100 point scale for the 10 questions, we simply need to sum the score for each question and multiply that number by 2.5. One great aspect of the SUS is its ability to give you a simple single usability score that you can compare across products and systems (your competitors, best of breed solutions, your old website, etc.).

Scoring

But how do you know what score is sufficient? If you are greater than 50, 60, 70, or 80, are you at an acceptable level? There isn’t to much literature out there on that topic but the folks at the measuring usability blog have done some interesting analytics and comparisons across industry. We generally take a simple back of the envelope analysis which suggests that if you were to receive 3 points on every question (that is, above the median, but not perfect), you’d score a 75. We take this as a proxy for a job mostly well done. For general purpose websites, we like to see scores that hover around that point and generally 70 is a number that we like to accept.

You won’t necessarily get too much insight into what the drivers causing issues actually are with SUS, although in practice, it’s often times easy to know where to start. We’ll often use SUS as a filter to let us know whether we need to dig deeper into all the tape, heatmaps, and screens that we captured during detailed usability studies. We talk about details in our next post.