Something about search engine crawler
 

A crawler reads the content from a page and usually stores this in a search engines database, called an index. From this index the search results are extracted when one has entered a keyword in a search engine. A crawler is also referred to as a robot or bot from a search engine. There are also...