web spider is an automated and methodological program that crawl the internet, programmed for task specific purpose and so is built in that specific way.
Some web spiders are developed to find email address in web pages also known as Email harvesting Crawlers, to check links in web pages, to find inactive links, validating links, validating HTML.
Spider is also developed to find the content from web pages and generate statistics or create copy of content from web pages. Especially search engines use web crawling to provide up-to-date.
Web crawler History: http://www.thinkpink.com/bp/WebCrawler/History.html
No comments:
Post a Comment
Feel free to disagree with the contents but keep it polite, please!
The blogger reserves the right to moderate any abusive comment(s).