Usability Testing: An Integral Part of Corporate Responsive Redesign

Redesigning Your Site to Be Responsive? You Need to Conduct Usability Testing. Often.

A vital component to our customer web site redesign at AEP is iterative usability testing. Our site is large – hundreds of pages, 7 sections, tons of content. Our analytics reveal that around 95% of our customer traffic is there to do one of 25 key tasks: view or pay their electric bill, report an outage or check outage status, or do some other account-related function. The other 5% of traffic is there to do anything else among our hundreds of pages. So, while our priority was to make the account-related functions prominent and easy to access, we couldn’t exactly kill all the other stuff.

As our information architecture and designing commenced, it became immediately imperative to continuously run usability studies on our wireframes and design comps – to make sure our design decisions were solid and informed by customer failures and successes.

We love the successes. But we rely on the failures to improve design.

Customer failures in our early designs led to 5 separate iterations of design-test-design, using Chalkmark for the testing. We observed a significant failure with a major task: reporting a power outage. Only around 40% of participants actually clicked in the right place – a section labeled Report Outages & Problems. Another one-third opted for the Contact Us tab, which we, as an electric utility, do not want. (We want customers to report outages through our automated web form, not a contact us form.)


In this early design, too many users were clicking Contact Us, and not Report Outages & Problems.

By the final iteration, we had created a design that worked and customers click on at least 85% of the time. But I shudder to think what the business impacts would have been had we NOT done usability testing.


By consolidating the outage-related functions, the most recent design attracts most of the users to the Outages & Problems area.

Comparative Testing
Another facet of our early testing was comparative usability testing, or A-B-C testing our site alongside 2 or 3 other electric utility sites. I cannot espouse the value of this enough. We gathered so much information about what works – and what does not work – in terms of design, labeling, and information architecture from our peer utilities. In many cases, our peers were vastly outperforming us in terms of consistency of where users clicked and how quickly they did so.


From our comparative study — we saw how much faster our peer utilities were for paying your bill.

Eight Rounds of Mobile Testing…and Counting
Taking our wide, deep site, and making it responsive invariably means making design decisions that affect layout, navigation, menus, touch interactions, and all the other nuances of presenting a web site to a small viewport.

Early on, we did a few rounds of in-person wireframe usability tests. At 320 pixels wide, the comps, when loaded on a test server, fit perfectly within the confines of an iPhone and certain Android phones. We experimented with various high-level elements, like the account login, search box, and navigation menu.


A very early mobile wireframe.

As our design became more refined, we began usability testing with full-color comps, created by our expert interaction designer, Nick Carron. Using Axure (wireframing software), I took Nick’s comps and created a pseudo-interactive experience. I then proceeded to conduct 5 remote, recorded mobile usability tests with representative users on iPhones and Android phones. Again, with each test, we patted ourselves for the successes. But lived for the failures.

While we’ve seen several issues with navigation and other design elements along the way (which we’re addressing with each subsequent round of test-design-test), I’ll point out a couple of key failures – with icons.

First, the menu navigation – aka “hamburger” nav. While this appears to be on its way to becoming a convention, it is not there yet – especially among mobile users who consider themselves to be novice to moderately experienced on mobile. While I don’t have anything close to statistical evidence for this statement, anecdotally, I observed customers in their 50s and older struggling with this more than younger users did. (That’s not to say younger ones didn’t occasionally have issues, or that older ones always did.)

What I know is that on tasks that required users to access the hamburger icon, a larger-than-expected percentage of them scrolled down the page – and in many cases, all the way down to the footer – where they found an appropriate link to tap. Recent studies and articles have actually shown this ineffectiveness: Aurora Bedford, of the Nielsen Norman Group, wrote an article on icon usability. Jennifer Aldrich blogged about it in her “User Experience Rocks” blog. Exis did a quantitative study on it.

Most users scrolled down — some all the way to the footer to find nav that was under the hamburger.

Simply adding the word “Menu” to the icon significantly improved the tapworthiness of it – although it’s much better at drawing in the Account-type functions than more general content like News or Save Energy.


Adding “Menu” to the hamburger icon helped a great deal. But it’s still not perfect.

The second icon failure was a simple [+] sign – which we used to show an expandable section on the page. Nope. People didn’t tap it. In fact, one person, who clearly saw it, commented that “plus means to add something, and that’s not what I want to do.” Duh.


Oops. Using [+] for expandable content wasn’t intuitive.

I live for those “duh” moments. The new design has a down arrow, but as of now, we still need to test its effectiveness.

There have been bunches of other fascinating findings from our mobile studies, which I won’t delve into in this blog:

  • Users will scroll a long page, but will bail out as soon as they see a link that might be remotely connected with what they’re looking for (even if it’s not).
  • Users are unforgiving about page load time on mobile — although slightly less unforgiving during a usability test.
  • Users don’t read content. They quickly scan.
  • Users will tap anything that looks like it could help them – even nonclickable headers.
  • Users have really big fingers compared to the things they have to tap.
  • Users learn your site during testing, so it’s best to re-order your tasks as you work through your list of participants.
  • Users will say something was totally perfect after they struggled for 2 minutes trying to find it. If they say: “So far, everything is easy, no problems here” or “OK, perfect” – DO NOT BELIEVE THEM or use what they say to add to your insights. Instead, rely on your observations.

Next Up – More Usability
Our next studies will be another mobile study, a desktop usability test (probably the first of 3 or 4), and an eye tracking study. In total, by the time the site launches late this fall, I expect we will have done more than a dozen in-person and remote usability studies – with input collected from more than 250 representative users.

Have you gone responsive yet? What kinds of usability activities did you do? And what amazing DUH moments have you had?


@ericdUX on Twitter
Connect on LinkedIn.

Security Questions Get Personal!

We’re all familiar with the traditional “mother’s maiden name” online account security question — one of the first I remember when security questions began to proliferate. To seemingly provide more secure, unguessable options, companies have added others, like “make and model of first car,” “street you lived on in 3rd grade,” and more questions whose answers don’t ever change. Of course, we also see examples of bad questions that DO change over time. These are mainly “favorites” types of questions: “favorite actor,” “favorite song,” “favorite movie,” “favorite teacher.” If you’re like me, my favorites change over time, rendering these types of questions useless. FYI, if you’ve ever had to reset your Apple ID password, you’ve noticed that Apple is among the biggest culprits of “favorites” questions.

Apple Security Questions

Apple security questions include many “favorites,” which change over time.

Favorites aren’t the only questions whose answers can change. How about “youngest child’s middle name?” Well, what if I have more children? Am I going to remember that fact a couple years down the line when I have forgotten my password, but increased my family count? Heck, questions about where you met your spouse can have ephemeral answers if you’re the divorcing type. Even questions like “nickname as a child” become difficult to answer. I had at least 4 nicknames that family and friends called me by. Which one do I choose? And what is the likelihood that 6 months later, I’ll remember the right nickname. My oldest cousin’s name? Well, it works for now, but what happens if he or she dies? (I actually refuse to enter questions like this, out of some strange superstition I feel…but that’s a whole separate issue.)

Not only do fungible answers confound the process, but arbitrary syntax rules as well. What if the first concert I attended was “U2,” but the site demands at least 3 characters in the field? If the first concert was Bruce Springsteen, do I include the space between the names? Does upper or lower case matter? If next year, while recovering my password, I type “bruce springsteen” when the site was expecting “Bruce Springsteen,” will it give me an error? What if I spell it “Springstein”? After multiple errors, I may start to doubt my own memory of whether that was my first concert — and go down the “Barry Manilow” path. Hypothetically speaking. Of course.

Usability problems of security questions aside, every now and then, I come across some gems that are worth capturing. These more, um, personal questions are both usable — for their answers can never change — and amusing. Seeing “What is the first name of the boy or girl that you first kissed?” from an online bank was a shocker for its uniqueness — and its being from a bank, of all places. (OK, admittedly, I spent a few seconds pondering if the girl who kissed me in first grade counted, or if the site was really after my first, shall we say, real kiss.)

Kissing Questions

Security questions get personal, with kissing questions!

This got me wondering — in companies’ efforts to come up with ever-increasing unguessable security questions — how much more personal might these questions start to get? Like, substitute “kissed” with, you know, other activities. Or maybe secret moles or other bodily anomalies that only you know the answer to? Or “How old were you when you lost your ________?” You fill in the blank. (I was thinking “first tooth.” Get your mind out of the gutter.) Regardless, I’m sure I’d glance back over my shoulder to make sure no one was watching as I typed the answers. And I’d probably never forget the answers or enter them wrong. Unless, of course, your site expects me to spell out “forty” instead of “40.”

If you see any clever security questions in your web travels, please pass them along.


Hyatt at Fisherman’s Wharf: A Beautiful Calamity of Uselessness

It’s obvious as soon as you enter the guest room at the Hyatt Fisherman’s Wharf that they’ve spent a lot of money making the rooms modern and sleek. Flat screen TV, stylish furniture, aesthetic bathroom, even an iHome iPod clock radio all contribute to a visual delight, sure to appease the well traveled or weary guest seeking the comforts of a modern home.

But when you actually start DOING things in the room, you quickly find it’s a case of design for its own sake and not for meeting people’s needs or goals. Continue reading

Usability Testing Vendor Products

Author’s Note: This blog was originally posted on March 30, 2011, on a company Facebook page. Now that I have a WordPress presence, I thought I’d post it here. Plus, there’s some good news at the end.

This blog posting looks at some recent examples of usability testing of vendor products, the challenges and resistance we faced from the vendor, and the lessons we have learned about why it’s valuable to conduct vendor testing while the project stakeholders observe live.

Continue reading