I am currently using the Common Crawl dump of web crawl data. Are their any other good providers of free web crawling data that I can blend with my Common Crawl data?
asked May 29, 2013 at 21:41
-
Are you looking for any specific kind of data? Or, in other words, why are you looking to extend the Common Crawl dump with data from other sources?– Patrick HoeflerMay 30, 2013 at 20:44
-
Yes, there are URLs in my training set of data that are not found in Common Crawl, so I would like to supplement common crawl data with any other sources.– Dan Ciborowski - MSFTMay 31, 2013 at 4:14
Add a comment
|
2 Answers
If the crawled data doesn't need to be very recent, the Internet Archive provides 80 terabytes of archived web crawl data from 2011 for research. Unfortunately, they don't say under which license they release the data, so it might not be Open Data as defined by the Open Definition.
answered May 30, 2013 at 20:55
-
I would prefer recent and one that is kept up to date like common crawl. Jun 1, 2013 at 8:53
I'm pretty sure there aren't, but your best bet would probably be to ask on the Common Crawl mailing list. That's a place where people who are interested in this kind of stuff hang out and it's low enough traffic that a polite request wouldn't be out of line -- the answer would probably be of interest to the entire community there.
