Human-Computer Interaction meets Economics: How to Measure Interface Utility with Mechanical Turkers

Google Tech Talk (more info below) June 6, 2011 Presented by Michael Toomim and Claus C Pörtner. Joint work with James A. Landay (all from U. Washington) ABSTRACT In 1983, Donald Norman presented the very first paper at the very first meeting of the Computer-Human Interaction (CHI) conference, and posed a still open problem: can we scientifically measure and optimize "user satisfaction"? Specifically, can we predict which interfaces users will value and choose to use, before building and deploying a product? Twenty-eight years later, we have yet to find objective measures of an interface's "value" or "user satisfaction." We present a new method to measure and quantify user choice based on Economics. To understand user behavior we run economic experiments with thousands of humans on the web. We pay users different amounts of money to use different interfaces that accomplish the same task, and determine the monetary amount that an interface variation is worth. For instance, we find that aesthetic changes to an interface significantly changes the amount of money we have to pay. Thus, our approach provides direct measures of user behavior and how much improvements in interfaces are worth to different types of users. This will help designers understand the trade-offs inherent in interface design. In addition, our methods allows for experiments that were previously not possible in both HCI and Economics. In this talk we explain our method, initial experiments, planned extensions, and the implications for both HCI and Economics.