Web crawler
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.
- Yahoo! Slurp is the name of the Yahoo Search crawler.
- Bingbot is the name of Microsoft's Bing webcrawler. It replaced Msnbot.
- FAST Crawler is a distributed crawler, used by Fast Search & Transfer, and a general description of its architecture is available
- Googlebot is described in some detail, but the reference is only about an early version of its architecture, which was based in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.
Open-source crawlers
- Aspseek is a crawler, indexer and a search engine written in C++ and licensed under the GPL
- DataparkSearch is a crawler and search engine released under the GNU General Public License.
