Saturday, September 11, 2010

Very interesting technology behind Google's new Instant Search

Anyone using Google search and who is paying attention has noticed the very different end-user experience. Showing search results while typing queries now requires that Google has to to generate at least 5 times the number of results pages, use new Javascript support for fast rendering of instant search results, and, most interesting to me, a new approach to their backend processing:

It has been about 7 years since I read the original papers on Google's Big Table and map reduce, so it is not at all surprising to me that Google re-worked their web indexing and search. The new approach using Caffeine forgoes the old approach of batch map reduce processing and maintains a large database that I think is based on Big Table and now performs continuous incremental updates.

I am sure that Google will release technical papers on Caffeine - I can't wait!

2 comments:

Tim said...

It looks like they're set to present next month at USENIX OSDI '10 http://www.usenix.org/events/osdi10/tech/ in Vancouver:

"Incremental Processing of Large Data Sets
Daniel Peng and Frank Dabek, Google, Inc."

Mark Watson, author and consultant said...

Hello Tim. That is interesting! Hopefully a long paper will be release soon.