Indexing the Web—It’s Not Just Google’s Business
http://www.alistapart.com/articles/indexing-the-web-its-not-just-googles-business/

Mullican, Lyle
List Apart, A
2009

Abstract:

Web databases do much more than passively store information. Part of their power comes from indexing records efficiently. An index serves as a map, identifying the precise location of a small piece of data in a much larger pile. For example, when I search for “web development,” Google identifies two hundred million results and displays the first ten—in a quarter of a second. But Google isn’t loading every one of those pages and scanning their contents when I perform my search: they’ve analyzed the pages ahead of time and matched my search terms against an index that only references the original content.