7

I am currently using the Common Crawl dump of web crawl data. Are their any other good providers of free web crawling data that I can blend with my Common Crawl data?

  • Are you looking for any specific kind of data? Or, in other words, why are you looking to extend the Common Crawl dump with data from other sources? – Patrick Hoefler May 30 '13 at 20:44
  • Yes, there are URLs in my training set of data that are not found in Common Crawl, so I would like to supplement common crawl data with any other sources. – Dan Ciborowski - MSFT May 31 '13 at 4:14
7

If the crawled data doesn't need to be very recent, the Internet Archive provides 80 terabytes of archived web crawl data from 2011 for research. Unfortunately, they don't say under which license they release the data, so it might not be Open Data as defined by the Open Definition.

  • I would prefer recent and one that is kept up to date like common crawl. – Dan Ciborowski - MSFT Jun 1 '13 at 8:53
2

I'm pretty sure there aren't, but your best bet would probably be to ask on the Common Crawl mailing list. That's a place where people who are interested in this kind of stuff hang out and it's low enough traffic that a polite request wouldn't be out of line -- the answer would probably be of interest to the entire community there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.