A web spider is an automated program which explores the web with the main purpose of discovering new content, and caching a copy of it so as to allow for later indexing and searching of that data on a search engine.
Although search engines are the main users of web spiders (sometimes also called Web Crawlers) they are often used for other purposes. One might spider a site in order to find broken links and other site errors for example.