The EchoUser Experience

During a recent project, a client asked us to evaluate the initial user experience for setting up a telephony product. They were not interested in doing a more traditional usability test as they felt that this “out of the box” experience could not be easily captured or replicated by forcing a user to perform a set of pre-defined tasks. They wanted feedback that was more contextual and ethnographic in nature, but they still wanted the quantitative data such as times, errors, assists, and usability ratings that you get from standard usability testing. The more we thought about this, the more we came to realize that there is a gap in methods from where lab testing ends and contextual/ethnographic studies begin.

Bridging the gap - Event Style Testing

Usability testing is often used for getting feedback or benchmarking the usability of key task flows of a product or service. These key tasks are discrete and isolated to make it easier to get more quantitative data, but this comes at the cost of losing what the natural workflow may be since a structure or order is enforced. The natural workflow is something that is researched through more contextual methods. What we wanted for this project was to be able to allow a natural workflow that would allow users to use multiple tools and do things that were not expected by the product team while being able to get metrics such as time, errors, assists, and perceived usability. With this in mind, we created a method that we’ve dubbed internally as Event Style Usability Testing to help bridge this gap between usability testing and contextual/ethnographic studies.

Components of Event Style Usability Testing

There are a couple of basic components to the use of Event Style Usability Testing

• A set of pre-defined “events” are identified that users are asked to perform.
• Users are allowed to use a system (or software) to do the events in any way and any order that they deem appropriate. There is no forced order.
• The test facilitator records the events as they take place creating a workflow record as well as keeping metrics such as time that events start and stop, number of errors and assists for events, and perceived usability of events.
• New events that were not pre-defined or expected are added ad hoc as they are observed during sessions.

To better understand this, what do I mean by an “event”? Events are very similar to tasks that are used in more standard usability testing with a couple of distinctions. Like tasks, events are created that users are asked to perform or experience. For example, an event would be to send an online payment to a person from an online bank account. If this were simply a task in a usability study, then users may be told specifics about who to send the payment to, for how much, and when it should arrive by. For an event, things are turned into a scenario as much as possible.  They are told what they are trying to accomplish, not given specifics.  In the case of the event for sending an online payment, users would choose who to send it to, for how much, from what account, etc.  Allowing them to shape the event using their own data or specific to their own interests allows interesting insights and observations about what users may do in real situations.

Once the events have been pre-defined, an event list is created for participants to use for the session. Prior to beginning the session, users are asked to read the entire list of events that they will be asked to do. They are then instructed to complete the list of events in whatever order they would like and makes the most sense to them. Having them read through the entire list of events this way allows them simulate more closely what they would do out in the real world. It’s assumed in most systems and software that users generally know what they want to do (or else why are they using the system or software in the first place) but may not know specifically how to do it. Event style testing aims to more closely replicated this.

The rest of the process now relies on the careful observation of the testing facilitator. Since there is no specified order that participants are using to complete events, the facilitator needs to monitor what events the participant is attempting or in the process of doing. This includes capturing time stamps of when an event is started, completed, or stopped and not completed, or resumed after being stopped previously. While observing each event, other usability metrics can be captured as well such as any observed errors, assists needed, and other usability scores.

The last piece to the Event Style Process is being flexible to add new events to record and capture in the middle of a session. When trying to replicate real world workflows, it can be nearly impossible to predict and plan for all paths and avenues that users may try to use to work through events. Also, unforeseen or unanticipated events may occur in context that would not occur in a highly structured usability test. The event style testing allows these newly observed events to be captured mid test so that they can be reported on later.

Benefits of Event Style Testing

The event style testing allows for a number of benefits and deliverables that cannot normally be obtained in a usability test. These include:

Workflow Analysis

Since the order that events take place is recorded during the study, a full task flow can be created and analyzed. This is particularly useful for systems or software where very little is known about how users setup or experience them, especially in complex systems where multiple tools or software may be needed. The resulting task flows can show where various tools and software are frequently used, in what order, and for what purpose. In the case of the telephony product we tested, the task flows highlighted at what points different software products were used for configurations and at what points users stopped to verify correct setups and troubleshoot problems.

Event Style Workflow

Quantitative data collection

While the natural workflow analysis can be observed, quantitative data such as errors, assists, and usability ratings can still be collected and compared across users. This creates a more holistic view of the experience being evaluated from a qualitative and quantitative standpoint, the best of both worlds.

Event Style Usability Testing has become a powerful method in our user experience toolbox to bridge the usability test and ethnographic research divide. We are always looking for new takes on old methods and hope others may find this useful as well!

Reinventing the wheel

I was recently chatting with my colleague Etan about how it’s ludicrous that some of the bigger tech companies in Silicon Valley (and beyond) build all software in-house. This means that while everything from file sharing, to collaboration, to idea generation and money tracking are all viable web-based services, tech monoliths basically take it upon themselves to reinvent the wheel, time and again.

Think back 15 years or so, and there were definitely good reasons for all this: firstly, many of the highly useful online services simply didn’t exist back in the day. No doing your taxes online with Turbo Tax, no chatting with your colleagues via Yammer, and certainly no checking in with Foursquare. The same applies to back-end magic, too: no WordPress for blogging, no Drupal or Joomla for content management, and no Ruby on Rails for rapid webapp development. So it’s definitely understandable that as different needs arose,  companies took it upon themselves to step in, foot the bill, and very often push the envelope.

App ubiquity

But no longer. We’re in the 21st century, people, and every need that can be conceived of has a matching web service or tool out in the ether. Yes, there really is an app for that.

Which is why I find it so ridiculous that companies are clinging to legacy file sharing systems, totally clunky conference call services, and outdated and burdensome security protocols. In an age where two guys in a garage can make Goliath-slaying software, it simply doesn’t make sense to hold on to old - and let’s face it, expensive - tools when better options are out in the wild. There’s no need to reinvent the wheel when evolution is just as good, and freely available.

Acquire me this

Let’s be fair to the big guys, though. Many companies do identify a need and end up buying a smaller company (or two) that fills it. This is good thinking: why invest time and energy when someone else has already done it? Everyone from Amazon to Cisco to Google to Intuit to Schlumberger to Yahoo does this, often to great success - but often ending up with a sprawling network of badly integrated portfolio companies and a more confused consumer offering.

When integration works, it’s great: I still use Turbo Tax and couldn’t give a hoot that it’s now owned by Intuit (:: yawn ::). Similarly, Flickr still runs well under Yahoo, and Zappos hasn’t visibly changed post-Amazon takeover.  When it doesn’t work, though, it can be a disaster, and the oil industry (aka Schlumberger, BP and others) is a case in point: I know for a fact that oil analysts often have to deal with dozens of data streams coming from different sources, often badly integrated, most oftentimes not.  And while it’s easy to point a finger at big oil, other industries suffer from this bloatedness as well: certain film industry companies suffer from 6+ month training regimes for new employees, simply because the range of different, clunkily-integrated tools that need to be learned is so vast. Definitely not efficient.

Security? Whatever.

Another main reason I hear that companies build stuff in-house - even buying companies and rebuilding their product offering from scratch - revolves around security. Data security, uptime security, offline security, web security; you name it, the reason for not just using an outside tool exists. But here’s the thing: it’s not like we aren’t already using web-based tools to transmit, monitor and manage sensitive information, because we are. Turbo Tax knows all about my 2009 income, Mint has a window into everything I own, and HighRise has a finger on the pulse of my deal flow. So is there some argument that my individual security needs are not the same (or as important) as those of a company on the Fortune 100 list?

Puh-lease.

Centralization and issues of scale (the most common big company needs) made sense in a time when hardware was scarce and security protocols minimal. But now the opposite is true: hardware is free, information is totally decentralized via the web, and it takes no more effort to secure one person’s information than it does 10,000. Sure, sometimes logging in to multiple online services can be a pain (and assumes you’re online to begin with), but it’s not that big of a drag - is it?

Organizations large and small would do well to jump on the webapp bandwagon and prevent the needless reinvention of the wheel. Evolution is where it’s at.

A last minute glitch and I could not get international roaming activated on my cell phone. This was the first time I was going to be without a convenient access to a phone while on an international trip. I wasn’t happy…

First stop Hong Kong airport, and I was pleasantly surprised to find the complete airport WiFi enabled. I was able to use Skype on my iPhone to talk back home, check my emails and send updates to my Facebook and Twitter accounts. So far, so good…

My entire stay in San Francisco, I must admit I never missed not having cellular connection…it seemed like the entire city was WiFi enabled even though it was only in parts. Any restaurant, shop, pub I found myself in had WiFi so staying connected was a breeze. In all my years in the US and my trips there over the last 3 years, I never saw WiFi so entrenched in the ecosystem until now.

It goes without saying that WiFi enabled Internet access on mobile phones is the way of the future (Virgin America has even started offering WiFi on their domestic flights).

Suddenly, with broadband data access, 3G seemed a bit too slow, cumbersome and a hindrance to the overall user experience.

In almost all aspects of consumer consumption behaviors in India, there has been a leapfrog like trend where the Indian consumer has bypassed some of the technical/behavioral aspects of adoption due to the late entry of some technology or product. Jumping from having no phones to the cell phone, getting introduced to the Internet directly on the mobile are some classic examples.

Apart from the government run service providers, 3G in India is still some distance away. Can India leapfrog in this aspect of adoption as well? Can we jump directly from Edge to WiFi? One can argue that carriers like TTSL, Reliance and Airtel are well positioned with their already existing home based broadband service to provide blanket (to some extent) WiFi coverage in metro cities. This could earn them enhanced revenue from their broadband service and also help bypass 3G. They could focus on mid/high tier WiFi enabled mobile phones, and generate greater ARPU through a much enhanced user experience of their VAS services.

It will be worth investigating a two-tier strategy - A limited 3G rollout to cover the rural geography in India for enhanced voice/data services and WiFi rollout in metro cities for data services.

Sometimes I come across positively delightful tidbits as I trawl the web, and this one, the “coming soon” page for Hosteeo, totally got me.

Nice look and feel? Check.

Mystery? Check.

Sense of being privy to something secret? Check.

Provides me with a way of finding out that secret? Check again.

All in all, a pretty effective way of getting my email address - but more importantly, my attention. Thanks to Chris Spooner for finding this gem.

A recent post by the folks at Nielsen from their AlertBox service titled “Fresh vs. Familiar: How Aggressively to Redesign” talks about how users tend to prefer incremental UI changes (i.e. familiarity) to something novel and unique.

The theory goes that a more aggressive redesign will force a user to relearn and re-familiarize herself with a new visual layout and navigation, thereby reducing the overall usability of the UI in question.  I think that in principle this makes sense - assuming we still live in the 1990s!

I can’t believe how traditional usability seems to design for the lowest common denominator user, assuming that any little change, anything that will make a user have to - god forbid - actually modify their behavior (gasp!) is a bad thing.  I call this the “1990s approach” to usability, because back in the 90s it made sense to design for novice users: the interwebs was barely hitting the road (and not yet the mainstream), and web designers the world over were just starting to figure out how to design websites and UIs that didn’t confuse people with conflicting colors, black backgrounds, and hideous layouts.

But we’re in the 21st century, people.

This means that for the first time in UI design’s short history we are able to create designs that delight while also serving a broader functional purpose, rather than worry too much about making something purely functional and usable. Sure, there’s a place for functional design that hits all the right usability buttons, but I get frustrated when I see the usability profession, my profession, getting pigeonholed as stodgy and uptight.

A friend of mine who used to work at Apple, someone who I thought understood that usability involves both function and (sometimes beautiful) form, drove it home for me: “Usability? That’s boring, there’s no creativity or design innovation in that. You guys just make sure things are usable.”

This isn’t because he doesn’t understand usability, and more serves to illustrate that usability overall has a slightly stale rap - one that I aim to change.

Who’s with me?

Measure me this

Over the past few years, the buzz on the street in the development space (read: international development), has been all about impact.  What is it? How does it affect our bottom line? Is it related to success - and if so, how? Who’s involved? How to improve it? But most importantly:

How to measure it…?

Quite a few people have been thinking about this for awhile now, and first among them is Sasha Dichter from Acumen Fund.  His thoughts on the matter merely reflect a broader movement within philanthropy that is focusing on what it means to create a sustainable development space that is responsive to changes on the ground.  Of course, before measurement comes definition, and what a bugbear that has proved to be!  Sara Olsen at the SVT Group has spent years trying to answer this question, and while she has been generally successful in some ways, it seems that a cookie cutter solution will be hard to come by.

Read more…

Since the launch of the MotoCliq yesterday, my Facebook and Twitter have been flooded with hurrahs and comments about this cool new device. Seven years in Motorola gave me enough friends there and some of them were heavily involved in making this Motorola dream a reality. Great work guys!

The iPhone kick-started it all and now Motorola has joined the social networking bandwagon (many others have but very few deliver the experience worth mentioning) with it’s first Android phone. They have gone a step ahead (as should be expected) and created a great experience around integration of the various social networking sites in a very cool looking ‘Homepage Dashboard’. Sanjay Jha in his talk at Mobilize 09 said that ‘one-to-one’ communication is a thing of the past. Now, it’s all about ‘one-to-many’ and the MotoCliq is true to this tenet.

This is all good but what happens if ‘Facebook’ or ‘Twitter’ are not ‘THE’ social networking sites anymore. The probability of that happening is low at least till the launch of Motocliq but same time next year and the story might be very different.

If changes in the human behavior at the high tier segment has forced companies to respond with the now social networking solutions, can they not extrapolate this behavior to the many many more low/mid tier user groups, provide customized social networking solutions, encourage this networking behavior and drive ARPU.

How do mobile phone companies respond to 2 basic questions?

• How do we keep pace with the ever-changing landscape of social networking sites
• How do we scale it so that such services are made available to the rest of the 80-90% low to mid tier cell phone users worldwide

Living in India the second question intrigues me more. For sometime now I have been thinking about a flexible mobile social networking platform that can be customized for different and specific user groups at the low and mid tier user categories.

Consider these user groups:

• Auto rickshaw/Taxi drivers across a city like Mumbai
• Truck drivers across India
• The millions of maids employed in every middle-class family of India
• The millions of farmers across various states in India

Now a few facts about the above user groups:

• They all have cell phones
• They do not possess an internet connection
• The cell phone is their main channel of communication and entertainment
• Their awareness levels of things around is very minimal and hence are an exploited lot
• They come from very specific user groups and normally communicate within their own group/community

With the above facts, a dashboard like networking application can improve the quality and extent of communication and also make them more aware, entertained and connected. It could eventually provide them with a better life.

I hope the mobile industry is already working on such solutions but I suspect the focus is very heavily bent on making the iPhone competitor. There is an opportunity waiting to be tapped. Any takers!

I stumbled across the work of Jonathan Jarvis (@JonathanJarvis) the other day, thanks to my housemate and colleague, @zaqintosh.  The idea of creating a “design language”, as Jarvis calls it, certainly isn’t new. Visualization geniuses like @Stamen (and here), Hans Rosling, Dan Roam and many others have long argued that a visual language is absolutely necessary when it comes to understanding complex information systems.  Even when I was at Origo, and we instituted a company-wide policy to take mind-map style notes and meeting sketches, a few of us took to creating our own “visual vocabularies” (as we called them) to help systematize our visual note taking, making them easily understood by anyone.

Goodbye chicken scratch, hello iconography and flow diagrams.

I think the most compelling piece of Jarvis’ story isn’t that we need to create a malleable visual taxonomy - systematizing is a natural step in any language, and visualization is no different.  No, the best part of his argument is that a new class of professionals, “New Mediators” as he calls them, will come to supplant - or blend - the previously separate roles of Journalist, Analyst and Designer.

I’m curious to see how this plays out - now I’m off to brush up on my After Effects skills, since it looks like I’ll be needing them.

(image courtesy of Jonathan Jarvis at www.newmediators.com)