International Usability Testing: some common challenges

For those who aren't familiar with usability testing, the idea is to have real users of an application (or Web site, mobile app, etc.) interact with a prototype (often in the form of black-and-white wireframes rather than a fully designed interface) before beginning development. This allows us to identify flaws in the design while they can still be easily addressed, rather than during the development phase, when changes are expensive and time-consuming. Usability testing is a critical component of a truly user-centred design process.

Earlier this year I had the chance to conduct usability testing in two different countries for one of our long-term clients, the International Road Transport Union (www.iru.org).  We have been helping them redesign a Web-based application that enables road transportation companies around the world to file advance cargo "pre-declarations" with customs offices, speeding up and simplifying the transport of goods across borders, while keeping a high level of cargo and data security.

For those who aren't familiar with usability testing, the idea is to have real users of an application (or Web site, mobile app, etc.) interact with a prototype (often in the form of black-and-white "wireframes" rather than a fully designed interface) before beginning development. This allows us to identify flaws in the design while they can still be easily addressed, rather than during the development phase, when changes are expensive and time-consuming. Usability testing is a critical component of a truly "user-centred" design process.

Since the application is currently used in 26 different countries, it was important that we find users in several different countries to participate. Naturally, the choice of these particular countries raised some challenges that are common to many international usability testing efforts.

Logistics

One question that frequently arises regarding international usability testing is how many markets to test in. It would be prohibitively expensive and time consuming to test in every country, so a common approach is to divide the world into "regions" and select a representative country from each. Major corporations often divide the world into four regions: EMEA (Europe, Middle East, Africa), North America, South America, and APAC (Asia Pacific region). 

Another approach, the one we chose for this project, is to choose test regions according to the size of your user base - Turkey and Belarus were among the largest markets for our application, so we chose those countries for our tests. It also helped that the IRU works closely with local associations in those countries who were able to schedule sessions with real users of the system, provide facilities (meeting rooms, computers and monitors, coffee, etc.) and coordinate travel plans.

I've worked on projects for other clients who had no local presence at all - in those cases the best option is usually to partner with a local market research or usability firm. It costs more, but there's really no way around it if you need to test in a particular market.

Language

There are two major challenges that arise with regards to language:
a) translation of the prototype
b) interpreting the feedback from the usability test participants.

With regards to point a, it is particularly important that the translator have a deep understanding of the relevant domain terminology (in our case, related to certain treaties covering international customs guarantees).  We were lucky to have in-house translators working with us, and as a further pre-caution we asked members of the local associations to review the translations for accuracy.

Interpreting (point b) is a particular challenge when conducting international usability tests, as the best practice in test "moderation" is to create an environment similar to the real context of use. There isn't much that feels natural about having a test moderator speaking to you in a foreign language while the interpreter speaks into your other ear a moment later. For this reason, I would generally recommend using a local usability tester rather than a foreign moderator with an interpreter.

In our case, for logistical reasons, we were stuck with the moderator / interpreter model, and just had to make it work. We allowed a little extra time for this activity, and were again lucky enough to have bilingual members of our client's organization take on the interpreting role, ensuring that they had the vocabulary and domain knowledge to explain specialized concepts.

Politics and local customs

Access to the end users of an application is sometimes an issue. Local representatives can be sensitive about outsiders contacting their customers and clients, even for a seemingly innocuous purpose like testing a user interface, as there is always the possibility that their customers won't like what they see or hear coming from "headquarters."  Sometimes, regional managers also have their own ideas about the right design for an interface, and may be worried that the users will express a different preference. It can be a challenge to explain to the local leadership that capturing the end-user's input helps everyone make better design choices, and is not a challenge to their own authority.
Inviting users to participate in the design process can also be an opportunity to show them that the organization cares about their input and values their feedback.

In our case, the local representatives were happy to recruit participants for us from amongst their user base. In Turkey, we visited users at their offices around Istanbul, accompanied by an interpreter and a member of the Istanbul Chamber of Commerce, which meant there were certain formalities that had to happen before we could get down to business. Though this sort of thing can be frustrating if you are on a tight timeline, it is important to respect the local business culture in order to set the right tone. It can also help establish rapport with the users, which is an important element of good test moderation - you want the user to be comfortable sharing their feedback with you openly, and a bit of a chat over coffee beforehand can help create that trust.

In Belarus, the users came to us, traveling from all over the country to participate. This meant that we were testing in a less "realistic" environment (we used a conference room in Minsk), but it allowed us to get a broader sampling of users in a shorter timeframe. These sorts of tradeoffs are common in planning international studies.

Conclusions

Though it is difficult to generalize, in many years of conducting usability tests around the world, I haven't seen dramatic differences in the way people from different countries use applications or web sites. The same usability issues arise in Europe, Asia and the Americas; labels aren't clear enough, users get lost in complicated navigation schemes, or can't find what they are looking for.

Instead, I have found that it is the specific ways of doing business or accomplishing tasks that vary internationally. For example, many of the clients we visited in Turkey were large-scale operations that needed tools for batch uploads of customs data and integration with other business systems.  In Belarus, the required data fields on various forms were slightly different, meaning we had to design a flexible interface that would accommodate varying numbers of fields and inputs.  These findings aren't always usability issues, but in practice they help us decide which features to highlight in the interface and which ones to relegate to secondary status, ultimately contributing to a better overall experience for all users.

Digital news!

Are you as digitally addicted as we are? We can supply you with a regular dose of digital news. Simply sign-up, or click to Follow us online.

About the author

Ethan Maehl
Ethan Maehl

Ethan has over 17 years’ experience analysing business stakeholder and user needs, as well as designing applications for Fortune 500 clients in the US, Europe and Asia.

Formerly heading User Experience at companies such as Expedia and Razor Fish, Ethan has extensive experience with E-commerce, Mobile, Social Media and Enterprise Applications.