The EchoUser Experience

During a recent project, a client asked us to evaluate the initial user experience for setting up a telephony product. They were not interested in doing a more traditional usability test as they felt that this “out of the box” experience could not be easily captured or replicated by forcing a user to perform a set of pre-defined tasks. They wanted feedback that was more contextual and ethnographic in nature, but they still wanted the quantitative data such as times, errors, assists, and usability ratings that you get from standard usability testing. The more we thought about this, the more we came to realize that there is a gap in methods from where lab testing ends and contextual/ethnographic studies begin.

Bridging the gap - Event Style Testing

Usability testing is often used for getting feedback or benchmarking the usability of key task flows of a product or service. These key tasks are discrete and isolated to make it easier to get more quantitative data, but this comes at the cost of losing what the natural workflow may be since a structure or order is enforced. The natural workflow is something that is researched through more contextual methods. What we wanted for this project was to be able to allow a natural workflow that would allow users to use multiple tools and do things that were not expected by the product team while being able to get metrics such as time, errors, assists, and perceived usability. With this in mind, we created a method that we’ve dubbed internally as Event Style Usability Testing to help bridge this gap between usability testing and contextual/ethnographic studies.

Components of Event Style Usability Testing

There are a couple of basic components to the use of Event Style Usability Testing

• A set of pre-defined “events” are identified that users are asked to perform.
• Users are allowed to use a system (or software) to do the events in any way and any order that they deem appropriate. There is no forced order.
• The test facilitator records the events as they take place creating a workflow record as well as keeping metrics such as time that events start and stop, number of errors and assists for events, and perceived usability of events.
• New events that were not pre-defined or expected are added ad hoc as they are observed during sessions.

To better understand this, what do I mean by an “event”? Events are very similar to tasks that are used in more standard usability testing with a couple of distinctions. Like tasks, events are created that users are asked to perform or experience. For example, an event would be to send an online payment to a person from an online bank account. If this were simply a task in a usability study, then users may be told specifics about who to send the payment to, for how much, and when it should arrive by. For an event, things are turned into a scenario as much as possible.  They are told what they are trying to accomplish, not given specifics.  In the case of the event for sending an online payment, users would choose who to send it to, for how much, from what account, etc.  Allowing them to shape the event using their own data or specific to their own interests allows interesting insights and observations about what users may do in real situations.

Once the events have been pre-defined, an event list is created for participants to use for the session. Prior to beginning the session, users are asked to read the entire list of events that they will be asked to do. They are then instructed to complete the list of events in whatever order they would like and makes the most sense to them. Having them read through the entire list of events this way allows them simulate more closely what they would do out in the real world. It’s assumed in most systems and software that users generally know what they want to do (or else why are they using the system or software in the first place) but may not know specifically how to do it. Event style testing aims to more closely replicated this.

The rest of the process now relies on the careful observation of the testing facilitator. Since there is no specified order that participants are using to complete events, the facilitator needs to monitor what events the participant is attempting or in the process of doing. This includes capturing time stamps of when an event is started, completed, or stopped and not completed, or resumed after being stopped previously. While observing each event, other usability metrics can be captured as well such as any observed errors, assists needed, and other usability scores.

The last piece to the Event Style Process is being flexible to add new events to record and capture in the middle of a session. When trying to replicate real world workflows, it can be nearly impossible to predict and plan for all paths and avenues that users may try to use to work through events. Also, unforeseen or unanticipated events may occur in context that would not occur in a highly structured usability test. The event style testing allows these newly observed events to be captured mid test so that they can be reported on later.

Benefits of Event Style Testing

The event style testing allows for a number of benefits and deliverables that cannot normally be obtained in a usability test. These include:

Workflow Analysis

Since the order that events take place is recorded during the study, a full task flow can be created and analyzed. This is particularly useful for systems or software where very little is known about how users setup or experience them, especially in complex systems where multiple tools or software may be needed. The resulting task flows can show where various tools and software are frequently used, in what order, and for what purpose. In the case of the telephony product we tested, the task flows highlighted at what points different software products were used for configurations and at what points users stopped to verify correct setups and troubleshoot problems.

Event Style Workflow

Quantitative data collection

While the natural workflow analysis can be observed, quantitative data such as errors, assists, and usability ratings can still be collected and compared across users. This creates a more holistic view of the experience being evaluated from a qualitative and quantitative standpoint, the best of both worlds.

Event Style Usability Testing has become a powerful method in our user experience toolbox to bridge the usability test and ethnographic research divide. We are always looking for new takes on old methods and hope others may find this useful as well!

Good beginnings.

I’ve been following WalMart’s sustainability plans for awhile now, and I have to admit I was initially impressed. One of the world’s biggest companies actually making strides in sustainability and forcing its suppliers to come to heel?  The ripples of the Sustainable Product Index, I was sure, would spread far.

That was until I read more about their plan. It’s pretty simple really, and involves three steps:

1. Ask suppliers 15 simple questions about their supply chain and production practices.

2. Create a Sustainability consortium of universities, companies, NGOs and government to “develop a global database of information on the lifecycle of products…from raw materials to disposal.”

Ok, so far so good.  And finally:

3. To (and here I quote) “translate the information stored in the database into a simple tool that informs consumers about the sustainability of products.”

Right. Because consumers are inherently dumb, and the complicated supply chain information needs to be “translated” before we get to have access to it.

While I’m sure that the information is complicated, and that most people buying Kraft mac ‘n cheese, jelly donuts and frozen dinners don’t give a flying crap about supply chains, leaving the end user out of the equation until the very end is a huge mistake. WalMart may have made strides on its way to taking over American industry, but it committed the cardinal sin of standards creation: only including the end user, you know, the actual people who will use the standard on a daily basis, at the end of the process.

It’s not about the labeling, stupid.

I imagine WalMart’s thinking has gone something like this:

“We have about a million moving parts to coordinate with the Index, so we’ll tackle our suppliers first because they’re more complicated. Then we’ll make things seem more legitimate with a warm-and-fuzzy Consortium (I actually like this move, FYI). And finally, we’ll slap some pretty labeling system on everything so Gina from Shattunket, Georgia can choose the toilet paper that speaks to her conscience just as well as it does her bottom.”

But this is all backwards. Creating an index to be used by regular people isn’t just about making it legible or usable (although that’s definitely important and should be thought about up front anyway), it’s about making it relevant.

And the only way to do that is to include the end user - you, me, everyone - right smack at the beginning the whole process. Word.

I had a great chat with Johannes over at IDEO last week. Johannes is in charge of bringing some quantitative flavor to IDEO’s magic, and he’s up to some interesting stuff. After our morning coffee conversation, though, a few ideas stuck with me:

1. User churn

Me and the team at EchoUser have been getting more involved with start ups of late, helping them with usability, general user experience and the like. General start up concerns revolve around reducing “user churn”, or in other words, “how to keep users interested and encourage ongoing usage of our product/service.” It turns out that this is a sticky problem, especially when it comes to balancing the needs of new users (who might get “churned”) and “habitual” users who already use the product on a regular basis.

Our work inevitable hunts for what we’ve come to call “the aha moment”, or that one particular experience that helps turn a casual/new user into a habitual one. Of course, we all know it’s probably not one single experience that does the trick, but the simplification is helpful nonetheless.

In talking with Johannes I drew out what usage paths look like for a given product:

user churn paths

Ideally, you want users to take the path going up and to the right, because this means they’ve had their “aha moment” and have now become habitual users.  And for start ups looking to raise funding from VCs, being able to say “our usability work has shown we can improve user churn by X% because we can guarantee an aha moment” is a very good thing.

2. Switching

Then it was Johannes’ turn to pick up the pen. He drew this, which deals with a marketing concept called “Switching”:

User switching

Basically, switching refers to the idea that many people, when trying out a product or service, will tend to switch between different ones - this is represented by their degree of “loyalty” to a given product.  The general point is to try to get users to either extreme, either 100% loyal to one product, or 100% to the other (to me this is a little simplistic, given that it’s a zero-sum outcome, but more on that another time).

It’s interesting to note that the Churn graph is a loose inversion of the Switching graph: users who have yet to experience their “Aha!” moment are less likely to become 100% loyal to a particular product - and therefore more likely to switch (as well as churn).

We’re finding more and more that startups have to balance design changes that appeal to both habitual users and new ones, changes that try to straddle the divide between 100% loyal and “I’m not so sure just yet”-loyal.  The balancing act is an interesting one to say the least, and we’re using our usability special sauce to make it that much easier. More to come.

Does not compute image

This morning, while recruiting participants for an upcoming startup usability study, I stumbled upon a mythical creature I didn’t think existed. After chasing it through the underbrush of the internet to no avail, I eventually gave up, disgusted and disheartened, for the creature was gone - potentially forever.

And the creature in question? That rare breed of human who doesn’t have access to a DSL connection.

I know, I know, breathe in deeply - I didn’t think they existed, either. But it turns out that this kind of person is out there, and we have to plan for them accordingly.  More and more I find that people respond to my online recruitment ads, often for a remote study that needs screensharing and the like, without having the necessary technological hardware to actually complete the study. Like that story where from the band of monkeys eventually one will write Shakespeare, sometimes potential participants make it into the pool when they really shouldn’t.

It would be easy to blame the participant - I mean, who doesn’t have DSL access these days in the U.S., anyway? As it turns out, quite a few people don’t, so in the end it really is our bad as user researchers, not the participants. We need to make sure we’re designing studies - and recruitment procedures - that aren’t exclusionary from a technological perspective, even when we’re building a product or service that requires (relatively) advanced technology.

Examples of non-traditional recruitment abound: Etan once stood in a BART station for hours to recruit BART riders, while Mick loitered in a Best Buy (with permission of course) hoping to snag potential wireless hub buyers. The main problem, however, is that these methods, while often very effective, are time consuming - and expensive.

So how to recruit in an internet-ready world, when we’re not all on the same technological page?  Any thoughts? Examples?

I’m currently working on a usability and design prototyping project for a San Francisco-based iPhone app company (that shall, for now, remain unnamed).  So far it’s been fascinating, and lots of fun figuring out new ways to test the app, record the sessions, and integrate rapid design prototypes from week to week.

One of the more interesting parts of the project revolves around a diary study activity: we’re basically following half a dozen app users over a month to see how their experience with the app evolves, for better or worse.  I’m currently sending out 2 mini surveys a week, and have a shared “whiteboard” google doc where they can jot down any thoughts they have on the fly.

Experience Blogging

The neatest trick to the diary study, in my opinion, is the inclusion of what I’m calling “experience” blogging: basically, I’ve encouraged the participants to send screenshots of interesting moments they encounter while using the app by using the iPhone’s built in screen capture function (”On/Off” and “Main menu” simultaneous click).  I set up a dedicated photoblog on Posterous.com, and the participants basically send along their screenshots - which are automatically populated on the Posterous site.  The end result is a very neat live stream of app moments, sort of like the “pulse” of the app.

It seems like the native screen capture functionality of the iPhone makes this particularly easy, though I don’t know if any other phones do the same.

Has anyone else tried anything like this before?

You always hear about people doing A/B testing “out in the wild” (as well as examples of how it can be misused) but it’s rare that it is ever noticeable (which is kind of the point).

So when my colleague Aaron and I were working next to each other and happened to visit Salesforce.com at the same time, we were surprised to see the following:

and this…

Notice the difference? Someone at Salesforce is testing out whether people will sign up for the free trial (top menu, red button).

Does anyone have any other examples of A/B testing you’ve come across out in the wild? Would love to see examples.

Much of our work results in our clients having to make changes to an existing product or service.  It’s kind of the point of usability and design: unless everyone loves your product, careful research and testing will be sure to raise a few things that could be changed. Whether it’s the color of an icon or the entire product concept, design leaves no stone unturned.

One of the common refrains we hear from clients is this:

“But how will users know what we’ve changed??!”

(implication: they will hate it…)

We have lots of answers to that question, but here’s an example of my favorite (and perhaps the simplest) answer:

The lesson, it would seem, is to just tell them.

A recent post by the folks at Nielsen from their AlertBox service titled “Fresh vs. Familiar: How Aggressively to Redesign” talks about how users tend to prefer incremental UI changes (i.e. familiarity) to something novel and unique.

The theory goes that a more aggressive redesign will force a user to relearn and re-familiarize herself with a new visual layout and navigation, thereby reducing the overall usability of the UI in question.  I think that in principle this makes sense - assuming we still live in the 1990s!

I can’t believe how traditional usability seems to design for the lowest common denominator user, assuming that any little change, anything that will make a user have to - god forbid - actually modify their behavior (gasp!) is a bad thing.  I call this the “1990s approach” to usability, because back in the 90s it made sense to design for novice users: the interwebs was barely hitting the road (and not yet the mainstream), and web designers the world over were just starting to figure out how to design websites and UIs that didn’t confuse people with conflicting colors, black backgrounds, and hideous layouts.

But we’re in the 21st century, people.

This means that for the first time in UI design’s short history we are able to create designs that delight while also serving a broader functional purpose, rather than worry too much about making something purely functional and usable. Sure, there’s a place for functional design that hits all the right usability buttons, but I get frustrated when I see the usability profession, my profession, getting pigeonholed as stodgy and uptight.

A friend of mine who used to work at Apple, someone who I thought understood that usability involves both function and (sometimes beautiful) form, drove it home for me: “Usability? That’s boring, there’s no creativity or design innovation in that. You guys just make sure things are usable.”

This isn’t because he doesn’t understand usability, and more serves to illustrate that usability overall has a slightly stale rap - one that I aim to change.

Who’s with me?

One of the biggest gripes among usability professionals is that their client/boss waits until the end of a product or service’s development life cycle to start worrying about user experience.  This is like slapping a nice paint job on a jalopy, or building an entire stack of Jenga before checking to see if the table has a wobbly leg - in the long run it’ll bite you in the rear and all come crashing down (literally).

This is all very well when applied to a tangible product, a visible service, but what happens when it’s applied to something a little more fuzzy, say, like team building?

A friend was recently describing his attempts at building his team within a larger organization.  His goal, he explained, was to build out his team so he could then more effectively tackle larger organizational problems beyond his unit.  By hiring product managers, designers, marketers and the like, he was desperately hoping to buy himself the space - organizational and mental/emotional - to think about how to engage in larger scale change management.

To me, this seems like a dangerous game.  Sure, building out your team might buy you more time as you delegate to others, but with delegation comes responsibility, which itself takes time.  Indeed, much more time than we all initially think.  Furthermore, a bigger team makes for a bigger target within the organization, so there is the potential that any gains will be lost in the bigger organizational chess game.

My advice? Start with the connective tissue, the core usability and user experience. Don’t build out a team when the core is rotten, and risk creating a silo whose walls will need to be broken down anyway.  Start with the human-t0-human interactions that define the basis of the organization, no matter what size it becomes.  And don’t stop until you feel it in your gut.  Think you have it down? Tear it all down and start again, and again, and again.  Conversely, don’t drag it out for months - set the ball rolling and maintain momentum into the future.

Call it office “culture”, a philosophy, or whatever; slap guidelines into a pamphlet if you like, or conduct executive seminars during lunchtime; whatever it is, make sure you deal with the baseline usability and connective tissue first, and that you’re not just putting lipstick on a pig.