Yu Centrik

It is often necessary to use fairly complex tools to perform usability tests on voice interfaces (cf. Nexus[1], Database Systems Corp.[2]). One of our projects involved a touch-activated automated telephone information system. We required a test solution that was flexible, easy to build, and which provided a relatively natural simulation of a telephone call that would allow us to capture the call and a certain number of events during said call. It also had to be easy to modify and not too expensive.

We therefore developed a simple flash utility that was adapted to simulate a user’s phone call into a voice interface. It had the following characteristics:

1. Easy to use interface;
2. Not demanding: running this kind of application only requires an Internet explorer and a FLASH player;
3. Simple to modify: tests were iterative, so changes were made to the application’s structure after each test (three in all);
4. Allows the user to manage streaming video and audio. Works with video and audio recording software;
5. Helps collect and analyze data: for every call, the tool allowed us to capture the type of button pressed, the steps taken through the voice interface, the amount of time spent listening to messages, etc…  All of these data are critical for performing an in-depth usability analysis of any voice interface.
6. Easy to transfer to other voice interface applications.
7. Not too expensive: a FLASH developer developed this tool for us in two weeks.

Tool Description (Front End)

A simple screen reproduces a virtual telephone, which only the moderator can see and operate. Participants hear the instructions and press the buttons on the external numeric keypad to move on to the following instructions. Voice prompts are played on the computer’s audio port.
The moderator can record the participant’s name and surname, as well as the type of scenario and persona. The moderator launches the application; stops it whenever the participants are finished with their tasks; can interrupt the process at any moment during playback; and can replay any section.

During the call, the buttons the participants touch appear on the screen one after the other. Thus, the moderator sees in real time what steps the participants took. It also helps avoid errors in note-taking and compile the interface’s critical points quickly between participants.
Logs are also available to retrace the in-session keypad interactions: number of times zero was pressed, time spent inactive, etc…

Control System (Back End)

The interface’s structure management application assigns a telephone keypad key to each menu option. This is how the voice application’s menu path, options, and messages are created, how mp3 audio files are associated to them, and how errors, inactivity, redirects and message interruptions are accounted for.

Example:

A menu is created from left to right (see the figure above).

1. Click on ¨+¨ to create a new thread. Click on ¨-¨ to delete one. A “thread” is a menu, a message, a choice of languages, etc.

2. Name it and assign it a type (menu, message, etc.) according to its internally-defined nomenclature. The mp3 audio file (recorded in English or French) should have the same name. Assign it a time. This is the time within which a user should react (like three seconds, as in our example).

3. In the right-hand section, each key on the keypad is assigned a function and a response is defined for inactivity or error (ex: what sound to play if no key has been pressed within three seconds).
As you can see, the structure can be modified very quickly.

Conclusion

The results showed that FLASH, which is most commonly used for graphic interfaces, can also be handy for prototyping voice interfaces.
It can be used to develop a flexible application quickly, when you need to replicate a telephone call and to easily record the user’s experience.

The system has the added advantage of being transposable to other voice applications. We see the potential to use it in remote usability tests in the future, since FLASH makes it possible to develop interactive multimedia tools online and in local environments.

Still, the tool needs to be improved by linking up with a real telephone. This would most resemble a real call made from home.

Webliographic References
[1] Franz Neeser. Testing IVR Systems, White paper. Nexus Telecom AG, Switzerland. May 2005.
http://www.nexus- ag.com/fileadmin/documents/Whitepaper/White_Paper_IVR_Testing_
Nexus8610_Ed_2.2.pdf

[2] Database Systems Corp.

Source image
Laptop computer image: openclipart.org/media/files/aurium/4163

So, where do I start?  If you’ve been following my Twitter @jayvidyarthi, you know that last week I experienced an overload of knowledge transfer at the annual CHI conference in Boston.  I wanted to write a blog updating all of you on some of the fantastic methodologies, processes and technologies on the fringe of our field.  First, let me tell you a little about my read on the main theme of the conference.

Tying directly into the Closing Plenary delivered by Kees Overbeeke from Eindhoven, I observed the underlying theme of HCI and design today as focusing on the humanization of technology.  Computer systems no longer have to be cold and digital.  Instead, they can behave more organically, installing themselves into our lives in a natural way.  Kees described how our relationships with technology can even go beyond such seamless integration.  He presented a lamp which gradually loses power.  As one pets the lamp, as if it were a cat or dog, it comes back to life.  This was an example of technology which can appeal to our emotions and connect with our users in a more profound way (while also being environmentally friendly!).  The tradeoffs between a practical and useful lamp and a “pet” lamp are still to be explored, but Kees’ point of view undoubtedly inspires questions about the future directions of human-computer interaction.

Although not every researcher is ‘dreaming of the impossible’ like Kees, a broad survey of projects from various angles at CHI 2009 show that seamless integration is the underlying theme of our technological future.  I decided to summarize a few interesting projects for you to check out, but by no means is this a comprehensive summary.  Check out the official CHI program summary to get many more descriptions and references on the fringe of HCI.

Some Highlights

• Eric Gilbert’s work on building a predictive model for social networks allowing a system such as Facebook to automatically distinguish your close friends from your distant acquaintances based on your behaviour with the system.  A short summary on Eric’s blog
• John Zimmerman is building a strong design philosophy centered around creating products which promote self-identity.  An application of product attachment theory to design specifically for role transition and self enhancement.  Really fantastic stuff for the creative theorist.  John’s Research Description
• Sunny Consolvo is working on the Ubifit Garden, a wallpaper for your mobile phone which measures how much time you spend sedentary, walking, running, biking, etc., and grows flowers and plants to indicate your physical exertion.  The system serves as a subtle ambient measure of your own exercise which creates a motivating factor within your cellphone.  A Poster Describing the Ubitfit Garden
• Tovi Grossman et. al. at Autodesk are designing and testing a new approach to measure the learnability of an interface: the Question-Suggestion model.  By having a coach directly interact with users, the method extracts specific problems and provides a more precise measure of interface learnability.  Extremely valuable for any kind of longitudinal usability testing of complex interfaces.  Description of the Paper @ ACM
• David Geerts from Leuven presented well-justified heuristics for the implementation of social functionality in interactive TV systems.  Hopefully I’ll hear more from David when I head to Euro ITV 2009 in Belgium to present Yu Centrik’s interactive TV research findings.  David’s Presentation Slides (jump to slide 33)
• Sarah Diefenbach and Mark Hassenzahl presented a thought-provoking piece on the interaction of Beauty and Usability on consumer product choice.  The paper served as a launching point for an intense panel discussion including a tirade on the definition of beauty from Bill Buxton.  Very engaging stuff.  Sarah and Mark’s Thought-Provoking Paper
• Leanne Hirshfield from Robert Jacob’s lab at Tufts University presented some early work on using fNIR brain imaging to get a read on human syntactic and semantic workload.  The eventual fruits of this work could lead to interfaces which can adapt to the users’ cognitive load by reading bloodflow in the brain.  If this project starts to fly, interfaces could get very interesting.  A Description of the Adaptive Interfaces using fNIR Project
• Alexander Hoffman is working on TypeRight, a keyboard which uses language dictionaries to adjust the resistance of keys on the keyboard based on the probability of each letter being pressed.  I tried using the keyboard and barring some minor technical issues, it seems to have potential to help us avoid type-os and increase our overall typing speed.  The TypeRight Project
• David Frolich presented a case study of StoryBank, a mobile-phone and touch-screen based system allowing children in rural India to develop narrated slideshow videos.  Being a kind of “local, embodied youtube”, the interfaces involved no language, enabling use by illiterate children in the village.  The findings were quite surprising, and the videos made were very powerful.  David was intent on stressing how our technology can help the developing world.  The StoryBankHomepage

Again, if you enjoy this type of work, check out the CHI program summary for more quick descriptions!  If you want details and have access to ACM’s digital library, check out the full proceedings.

Our usability test lab has a software environment allowing for capture, recording, analysis and full interpretation of observable events during a usability test. Moreover, since we are very interested in user behaviour, verbal comments, facial expressions and eye movements, as well as keeping a record of keystrokes and mouse movement, we also use other software.

How is usability testing different when done on-site instead of remote? Here is a list to help you weigh the pros and cons of each method.

ON-SITE USABILITY TESTING

Pros

• Better performance for testing different interactive systems (web, software, mobility, interactive TV, vocal interfaces, etc.) and eye tracking techniques
• All participants are accomplishing the same tasks in the same environment, resulting in more stable performance data
• Technical set-up for the tests is easier
• By being on-site, the moderator can stay close to the participant and observe non-verbal behavior, hesitations and react quickly
• Audio and video, especially facial expressions are recorded
• The Client has the opportunity to observe users in real-time. It also allows them to take a break from their busy schedule to recharge and learn about their users first-hand

Cons

• Knowing he is being watched, the participant’s behavior might be influenced by a desire to perform well, despite assurances that they are not being tested
• Recruitment can be difficult because there are many marketing research companies asking for people to participate in surveys
• The pool of potential participants is limited to the local population otherwise, the usability experts need to travel
• Absenteeism rate is higher because participants have to travel to the site of the test. Other factors include weather, workload and last-minute cancellations after work, etc., might affect attendance)

REMOTE USABILITY TESTING

Pros

• The participant is in his natural habitat, at home or at work, in a real-life situation
• The technology allows for tests to be conducted anywhere in the world and in the participant’s first language
• The participant is more comfortable because he is working with familiar equipment or technology
• The participant seems less stressed, is acting more like his usual self, and is possibly less tolerant
• The moderator can still see everything the participant is doing on screen and hear what he is saying
• The moderator gets a glimpse of the participant’s private life through the use of shared screens
• The client can observe the tests from a remote location
• The sytem allows for a complete recording of screen interactions and audio
• Recruitment is easier because, having eliminated the traveling factor, pool is not limited to the local population

Cons

• Logistical elements are greater in terms of technical requirements. We need to do more pre-testing and use more internal resources
• Last minute or uncontrollable technical problems are more likely
• High speed bandwidth is essential to avoid long delays
• The use of a cellular phone for capturing audio is not ideal and the use of a hands-free landline is recommended
• Planning is more complicated because of timezones
• It’s harder for a participant to explain what he is doing or feeling when discussions between the moderator and the participant is in the participant’s second or third language
• It’s impossible to record non-verbal behavior

Either way, don’t stop testing your products with users, their feedback is too important to leave it out!

A usability testing protocol is a document that describes how to conduct usability testing. It’s also the spring board for discussions between Yu Centrik and client’s product teams.

The main goal of testing products with users is to identify usability issues encountered by users when using a product.

Through usability testing, Uability experts can observe how the user interacts with the product, evaluate how user-friendly it is and identify which functions were easier to use and which ones were more difficult.

Testing objectives are established at the onset and evaluated through scenarios developed with the Client.

During usability testing, participants are asked to do a number of specified tasks while explaining, out loud, exactly what they are doing as they accomplish the different scenarios. When the focus is on performance metrics then Retrospective Think Aloud (RTA) is best.

A user-profile defining the caracteristics of participants needed for testing is developed with the Client. Recruited participants should be representative of the product’s target client segment and include a sufficient number of people.

­ On the day of the test, the participant is welcomed into Yu Centrik’s offices.The moderaror explains the process and what they will be asked to do.
­
Each testing session lasts 60 minutes with the exception of eye tracking tests which can last 90 minutes.

The participant works through each task scenario individually and discuss as he goes along or after completing each scenario. The moderator listens and asks follow-up questions while an observer takes notes. Each session is filmed and client representatives observe in real-time.

Qualitative data is also gathered after each task in order to understand the motivations, perceptions and experience of the user. Data is then compiled, summarized and analysed.

A report on the test results, including an analysis of the results and recommendentations, is submitted and presented to the Client.

When possible, usability experts find it useful to follow-up on recommendations made because it gives them a chance to ensure that they were properly understood.

A lot of companies who wish to create or improve their web sites leave out the process of development of information architecture. Is this because of a lack of understanding of its principles and of the way it’s represented? Or is this because of the “middle of the road” aspect between the abstract ideas and concepts of a strategy and the more concrete wireframes which are closer to the final product?

What is important to know is that because information architecture bridges the gap between the abstract and the concrete, it is not only valuable, but crutial. Without it, it is impossible to build a product that represents the objectives of the company and at the same time satisfies the users’ needs. It would be virtually impossible for a product or service to thrive in competitive business without a well thought-out design structure.

Information architecture is applicable to whatever interactive product we design, be it a Web site, an interactive TV or a cell phone application. They are all dynamic products where content and features change with time. That’s the reason why a website’s basic structure must allow it to grow without significant changes that may cause a huge impact for final users and for those who keep the product alive. By the way, the biggest advantage of a well-designed information architecture is that it makes the product easier for those who use the final product and for those who are in charge of its management.

During a project, the time committed to define the information architecture must not be seen as wasted time but rather as time needed to create a high quality product in short and long terms.