GTAC 2011: How to Hit a Moving Target in the Cloud

6th Annual Google Test Automation Conference 2011 (GTAC 2011) "Cloudy With A Chance Of Tests" Computer History Museum Mountain View, CA USA October 26-27, 2011 Presented by Vishal Chowdhary, Microsoft ABSTRACT In this session, we share our experiences for testing the Microsoft Translator (MT) service. Testing the translator service can be divided into two broad areas -- (1) Testing of the machine learning system that returns the translation answer and (2) the web service that serves translate requests running on the Microsoft cloud. The MT service offers translation in 36 different languages for users across the globe through UI and API. Testing the MT cloud service without user data would be like shooting in the dark. From a product standpoint, we need to answer questions like How do we prioritize languages for improvement? How do we divide resources in our data center for effective capacity distribution for the languages? From a test perspective we need to understand how the system is being used so that we can ensure that any new bits deployed will hold-up in production. We will discuss the following three problems: Strategy for testing the machine learning system. How do you perform load testing for the translation service running in the cloud? How can one extend the role of test by completely skipping the 'opening bugs' step? With Microsoft for the past seven years, Vishal Chowdhary is currently with the MSR - Machine Translation (MT) team where he is responsible for both performance and scale testing of the MT cloud service and the machine translation quality. In the past, he has been involved with BizTalk, .NET Framework Wf and Wcf, and the Windows Azure AppFabric teams where he owned test strategy, test framework design, and feature testing. Vishal is passionate about making testing simpler and more interesting by employing new testing strategies and processes that prevent defects from entering the product.