A Web crawler is a computer that works it’s way through websites on the Internet following links and discovering new content. This new content is then organised and indexed with any links added to the ‘crawl frontier’ which represents future pages to be visited.
As you can imagine a huge amount of new data is added to the Internet constantly and existing indexed pages often change. It is important, therefore, for modern crawlers to be very selective and intelligent about which pages they visit and index in order to provide the best experience to search engine users.
Other uses for crawlers include webmasters checking their own sites for broken links and any errors in the HTML code that makes up the site and those who might look to send out SPAM will often crawl websites looking for e-mail addresses.