A spider (also known as a “crawler”) is a piece of software that is used to crawl the internet and index data appropriately.