Skip to main content

Web crawling is done typically for building and maintaining a Web Index. The crawled data is compared to previous versions of that dataset for changes. Web scraping is similar to web crawling, except that the site being crawled is scraped by the bot (crawler).

Web Crawling is a method of programatically assessing a web site's content, by systematically browsing the site via a bot, aka "crawling" for the purpose of Web Indexing. Bots, aka Spiders, aka FOAF Web Scutter, are scripts typically programmed to copy a sites content for the purpose of comparing it to the sites content at a future/paste date (Web Indexing). The differences are typically used for updating search engine result pages. Bots typically can validate hyperlinks, HTML, CSS, and JavaScript (Web Platform Technologies), empowering it to make decisions in regards to a sites links, content (seo), and functionality (ajax and accessibility support are two examples)