Robots.txt is a file webmasters can host on their server if they wish to instruct web robots (usually spidering robots such as Google bot) to take certain actions/avoid certain pages. This is to avoid spidering of duplicate content/private content or administrative areas of the site – all of which can have negative impacts.