The EchoUser Experience

During a recent project, a client asked us to evaluate the initial user experience for setting up a telephony product. They were not interested in doing a more traditional usability test as they felt that this “out of the box” experience could not be easily captured or replicated by forcing a user to perform a set of pre-defined tasks. They wanted feedback that was more contextual and ethnographic in nature, but they still wanted the quantitative data such as times, errors, assists, and usability ratings that you get from standard usability testing. The more we thought about this, the more we came to realize that there is a gap in methods from where lab testing ends and contextual/ethnographic studies begin.

Bridging the gap - Event Style Testing

Usability testing is often used for getting feedback or benchmarking the usability of key task flows of a product or service. These key tasks are discrete and isolated to make it easier to get more quantitative data, but this comes at the cost of losing what the natural workflow may be since a structure or order is enforced. The natural workflow is something that is researched through more contextual methods. What we wanted for this project was to be able to allow a natural workflow that would allow users to use multiple tools and do things that were not expected by the product team while being able to get metrics such as time, errors, assists, and perceived usability. With this in mind, we created a method that we’ve dubbed internally as Event Style Usability Testing to help bridge this gap between usability testing and contextual/ethnographic studies.

Components of Event Style Usability Testing

There are a couple of basic components to the use of Event Style Usability Testing

• A set of pre-defined “events” are identified that users are asked to perform.
• Users are allowed to use a system (or software) to do the events in any way and any order that they deem appropriate. There is no forced order.
• The test facilitator records the events as they take place creating a workflow record as well as keeping metrics such as time that events start and stop, number of errors and assists for events, and perceived usability of events.
• New events that were not pre-defined or expected are added ad hoc as they are observed during sessions.

To better understand this, what do I mean by an “event”? Events are very similar to tasks that are used in more standard usability testing with a couple of distinctions. Like tasks, events are created that users are asked to perform or experience. For example, an event would be to send an online payment to a person from an online bank account. If this were simply a task in a usability study, then users may be told specifics about who to send the payment to, for how much, and when it should arrive by. For an event, things are turned into a scenario as much as possible.  They are told what they are trying to accomplish, not given specifics.  In the case of the event for sending an online payment, users would choose who to send it to, for how much, from what account, etc.  Allowing them to shape the event using their own data or specific to their own interests allows interesting insights and observations about what users may do in real situations.

Once the events have been pre-defined, an event list is created for participants to use for the session. Prior to beginning the session, users are asked to read the entire list of events that they will be asked to do. They are then instructed to complete the list of events in whatever order they would like and makes the most sense to them. Having them read through the entire list of events this way allows them simulate more closely what they would do out in the real world. It’s assumed in most systems and software that users generally know what they want to do (or else why are they using the system or software in the first place) but may not know specifically how to do it. Event style testing aims to more closely replicated this.

The rest of the process now relies on the careful observation of the testing facilitator. Since there is no specified order that participants are using to complete events, the facilitator needs to monitor what events the participant is attempting or in the process of doing. This includes capturing time stamps of when an event is started, completed, or stopped and not completed, or resumed after being stopped previously. While observing each event, other usability metrics can be captured as well such as any observed errors, assists needed, and other usability scores.

The last piece to the Event Style Process is being flexible to add new events to record and capture in the middle of a session. When trying to replicate real world workflows, it can be nearly impossible to predict and plan for all paths and avenues that users may try to use to work through events. Also, unforeseen or unanticipated events may occur in context that would not occur in a highly structured usability test. The event style testing allows these newly observed events to be captured mid test so that they can be reported on later.

Benefits of Event Style Testing

The event style testing allows for a number of benefits and deliverables that cannot normally be obtained in a usability test. These include:

Workflow Analysis

Since the order that events take place is recorded during the study, a full task flow can be created and analyzed. This is particularly useful for systems or software where very little is known about how users setup or experience them, especially in complex systems where multiple tools or software may be needed. The resulting task flows can show where various tools and software are frequently used, in what order, and for what purpose. In the case of the telephony product we tested, the task flows highlighted at what points different software products were used for configurations and at what points users stopped to verify correct setups and troubleshoot problems.

Event Style Workflow

Quantitative data collection

While the natural workflow analysis can be observed, quantitative data such as errors, assists, and usability ratings can still be collected and compared across users. This creates a more holistic view of the experience being evaluated from a qualitative and quantitative standpoint, the best of both worlds.

Event Style Usability Testing has become a powerful method in our user experience toolbox to bridge the usability test and ethnographic research divide. We are always looking for new takes on old methods and hope others may find this useful as well!

One of the biggest gripes among usability professionals is that their client/boss waits until the end of a product or service’s development life cycle to start worrying about user experience.  This is like slapping a nice paint job on a jalopy, or building an entire stack of Jenga before checking to see if the table has a wobbly leg - in the long run it’ll bite you in the rear and all come crashing down (literally).

This is all very well when applied to a tangible product, a visible service, but what happens when it’s applied to something a little more fuzzy, say, like team building?

A friend was recently describing his attempts at building his team within a larger organization.  His goal, he explained, was to build out his team so he could then more effectively tackle larger organizational problems beyond his unit.  By hiring product managers, designers, marketers and the like, he was desperately hoping to buy himself the space - organizational and mental/emotional - to think about how to engage in larger scale change management.

To me, this seems like a dangerous game.  Sure, building out your team might buy you more time as you delegate to others, but with delegation comes responsibility, which itself takes time.  Indeed, much more time than we all initially think.  Furthermore, a bigger team makes for a bigger target within the organization, so there is the potential that any gains will be lost in the bigger organizational chess game.

My advice? Start with the connective tissue, the core usability and user experience. Don’t build out a team when the core is rotten, and risk creating a silo whose walls will need to be broken down anyway.  Start with the human-t0-human interactions that define the basis of the organization, no matter what size it becomes.  And don’t stop until you feel it in your gut.  Think you have it down? Tear it all down and start again, and again, and again.  Conversely, don’t drag it out for months - set the ball rolling and maintain momentum into the future.

Call it office “culture”, a philosophy, or whatever; slap guidelines into a pamphlet if you like, or conduct executive seminars during lunchtime; whatever it is, make sure you deal with the baseline usability and connective tissue first, and that you’re not just putting lipstick on a pig.

Lately I’ve been struck by seams.  Seams everywhere: sometimes in the right places, sometimes definitely in the wrong places, but almost always noticeable.  It’s been a while since I went through an experience where I didn’t notice the seams.

Seth points out that seams are important, and I think he’s right.  But like most things, you only really notice them when they don’t work, or are broken.

“Where’s the damn mute button on the remote?”

“Which of these icons does what I want?”

“How to I turns on the wipers for this car?”

Each of these isn’t a deal breaker in and of itself (unless not being able to turn the wipers on/down ends in a crash), but in the long run, like Chinese water torture, they add up. Drip, drip, drip, drip…

The Bolt || Peters crew point out that one of web site UX’s 10 biggest faux pas is to unnecessarily block access to content, creating a huge seam. I remember hitting a similar seam while trying to read a Globe and Mail article way back when, and was appalled at both A. the seam and B. the audacity of G&M’s assumption that in today’s world of free! free! free! I would consider paying anything to read the article - let alone a whopping 5 dollars!

It’s 7:00 a.m. and I am crammed into a bus packed with commuters.  As I try gingerly to prevent my face from rubbing against the armpit of an adjacent passenger, I awkwardly suspend my clipboard over my head and try to take notes on every aspect of the experience.  You see, while other passengers are traveling to their offices on this bus, for today the bus IS my office.

So why might you ask am I here? I’m working on my first project as a newly hired User Experience Researcher/Designer at EchoUser, and I am meeting my 3rd transportation study participant for BART and AC Transit.

Read more…

Welcome to The EchoUser Experience, where we hope to inform, entertain, and offer a window into our personality. We are designers, developers, usability engineers, business strategists, human factors specialists… surfers, hockey players, runners, and enjoyers of happy hours. We have many characteristics that make up who we are. So do the products and people we work with.

Read more…