Usability Testing: An Integral Part of Corporate Responsive Redesign

Redesigning Your Site to Be Responsive? You Need to Conduct Usability Testing. Often.

A vital component to our customer web site redesign at AEP is iterative usability testing. Our site is large – hundreds of pages, 7 sections, tons of content. Our analytics reveal that around 95% of our customer traffic is there to do one of 25 key tasks: view or pay their electric bill, report an outage or check outage status, or do some other account-related function. The other 5% of traffic is there to do anything else among our hundreds of pages. So, while our priority was to make the account-related functions prominent and easy to access, we couldn’t exactly kill all the other stuff.

As our information architecture and designing commenced, it became immediately imperative to continuously run usability studies on our wireframes and design comps – to make sure our design decisions were solid and informed by customer failures and successes.

We love the successes. But we rely on the failures to improve design.

Customer failures in our early designs led to 5 separate iterations of design-test-design, using Chalkmark for the testing. We observed a significant failure with a major task: reporting a power outage. Only around 40% of participants actually clicked in the right place – a section labeled Report Outages & Problems. Another one-third opted for the Contact Us tab, which we, as an electric utility, do not want. (We want customers to report outages through our automated web form, not a contact us form.)


In this early design, too many users were clicking Contact Us, and not Report Outages & Problems.

By the final iteration, we had created a design that worked and customers click on at least 85% of the time. But I shudder to think what the business impacts would have been had we NOT done usability testing.


By consolidating the outage-related functions, the most recent design attracts most of the users to the Outages & Problems area.

Comparative Testing
Another facet of our early testing was comparative usability testing, or A-B-C testing our site alongside 2 or 3 other electric utility sites. I cannot espouse the value of this enough. We gathered so much information about what works – and what does not work – in terms of design, labeling, and information architecture from our peer utilities. In many cases, our peers were vastly outperforming us in terms of consistency of where users clicked and how quickly they did so.


From our comparative study — we saw how much faster our peer utilities were for paying your bill.

Eight Rounds of Mobile Testing…and Counting
Taking our wide, deep site, and making it responsive invariably means making design decisions that affect layout, navigation, menus, touch interactions, and all the other nuances of presenting a web site to a small viewport.

Early on, we did a few rounds of in-person wireframe usability tests. At 320 pixels wide, the comps, when loaded on a test server, fit perfectly within the confines of an iPhone and certain Android phones. We experimented with various high-level elements, like the account login, search box, and navigation menu.


A very early mobile wireframe.

As our design became more refined, we began usability testing with full-color comps, created by our expert interaction designer, Nick Carron. Using Axure (wireframing software), I took Nick’s comps and created a pseudo-interactive experience. I then proceeded to conduct 5 remote, recorded mobile usability tests with representative users on iPhones and Android phones. Again, with each test, we patted ourselves for the successes. But lived for the failures.

While we’ve seen several issues with navigation and other design elements along the way (which we’re addressing with each subsequent round of test-design-test), I’ll point out a couple of key failures – with icons.

First, the menu navigation – aka “hamburger” nav. While this appears to be on its way to becoming a convention, it is not there yet – especially among mobile users who consider themselves to be novice to moderately experienced on mobile. While I don’t have anything close to statistical evidence for this statement, anecdotally, I observed customers in their 50s and older struggling with this more than younger users did. (That’s not to say younger ones didn’t occasionally have issues, or that older ones always did.)

What I know is that on tasks that required users to access the hamburger icon, a larger-than-expected percentage of them scrolled down the page – and in many cases, all the way down to the footer – where they found an appropriate link to tap. Recent studies and articles have actually shown this ineffectiveness: Aurora Bedford, of the Nielsen Norman Group, wrote an article on icon usability. Jennifer Aldrich blogged about it in her “User Experience Rocks” blog. Exis did a quantitative study on it.

Most users scrolled down — some all the way to the footer to find nav that was under the hamburger.

Simply adding the word “Menu” to the icon significantly improved the tapworthiness of it – although it’s much better at drawing in the Account-type functions than more general content like News or Save Energy.


Adding “Menu” to the hamburger icon helped a great deal. But it’s still not perfect.

The second icon failure was a simple [+] sign – which we used to show an expandable section on the page. Nope. People didn’t tap it. In fact, one person, who clearly saw it, commented that “plus means to add something, and that’s not what I want to do.” Duh.


Oops. Using [+] for expandable content wasn’t intuitive.

I live for those “duh” moments. The new design has a down arrow, but as of now, we still need to test its effectiveness.

There have been bunches of other fascinating findings from our mobile studies, which I won’t delve into in this blog:

  • Users will scroll a long page, but will bail out as soon as they see a link that might be remotely connected with what they’re looking for (even if it’s not).
  • Users are unforgiving about page load time on mobile — although slightly less unforgiving during a usability test.
  • Users don’t read content. They quickly scan.
  • Users will tap anything that looks like it could help them – even nonclickable headers.
  • Users have really big fingers compared to the things they have to tap.
  • Users learn your site during testing, so it’s best to re-order your tasks as you work through your list of participants.
  • Users will say something was totally perfect after they struggled for 2 minutes trying to find it. If they say: “So far, everything is easy, no problems here” or “OK, perfect” – DO NOT BELIEVE THEM or use what they say to add to your insights. Instead, rely on your observations.

Next Up – More Usability
Our next studies will be another mobile study, a desktop usability test (probably the first of 3 or 4), and an eye tracking study. In total, by the time the site launches late this fall, I expect we will have done more than a dozen in-person and remote usability studies – with input collected from more than 250 representative users.

Have you gone responsive yet? What kinds of usability activities did you do? And what amazing DUH moments have you had?


@ericdUX on Twitter
Connect on LinkedIn.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s