GTAC 2010: Early Test Feedback by Test Prioritisation

Google Test Automation Conference 2010 October 28-29, 2010 Early Test Feedback by Test Prioritisation Presented by Shin Yoo, University College London & Robert Nilsson, Google Inc. ABSTRACT As software become more complex and the number of clients grows, it becomes harder to modify legacy modules without breaking clients. Large scale regression testing takes a lot of resources and, if a shared version control system is used, detecting problems at the integration stage can affect the productivity of client teams. We present a testing approach, based on meta-heuristic optimisation, that empowers developers to select a set of pre-submit tests that, while testing as many integration points as possible, can be executed within manageable amount of time. Our assumption is that this will capture a large percentage of cross-module breakages while significantly decreasing the testing effort. The approach was implemented and evaluated in a series of experiments at Google; it sits between two extreme pre-submit testing practices currently used by Google developers: either run a small number of familiar (i.e in the same directory) tests or submit the code to TAP in order to execute semi-exhaustive collection of tests automatically. Empirical evaluation shows that the new approach can still effectively detect test breakage without executing relevant tests exhaustively. Shin Yoo is a postgraduate researcher working in Centre for Research on Evolution, Search and Testing (CREST) at University College London. His main expertise lies in application of meta-heuristic optimisation for various software engineering problems including the optimisation of regression testing process. He has published a survey of the state-of-the-art regression testing techniques recently. Robert Nilsson is a Software Engineer in Test working in the google Zurich office where he has been developing tools for automated testing and developer productivity since early 2007. He holds a Ph.D. in Computer Science from the university of Linkoping, his thesis specialized in mutation testing and automated test case generation for real-time systems. Slides for this presentation are available at https://docs.google.com/leaf?id=0AYfT-BFGDnQkZDVzcmhiNF8xMDdoY3RwY3BkbQ&sort=name&layout=list&pid=0B4fT-BFGDnQkYzVlMWY4M2EtYjRkYi00MDQ5LTg5ODUtNGQ4ZWU5N2NlMTRl&cindex=9