Random Sampling from a Search Engine's Index

Google Tech Talks August 17, 2006 Ziv Bar-Yossef joined Google from the Technion - Israel Institute of Technology in Haifa, Israel. He received his PhD from UC Berkeley in 2002, and was a Research Staff Member at the IBM Almaden Research Center prior to joining Technion. This was an academic study conducted before Ziv Bar-Yossef joined Google and does not represent Google's views. ABSTRACT We revisit a problem introduced by Bharat and Broder almost a decade ago: how to sample random pages from a search engine's index using only the search engine's public interface? In this paper we introduce two novel sampling techniques: a lexicon-based technique and a random walk technique. Our methods...