I am currently running a crawl of the internet using Nutch. This requires a list of URLs to start as a seed. I currently have a 500k url seed.
But I am looking for any open data sources to provide good starting seeds for web crawls.
Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It only takes a minute to sign up.Sign up to join this community
You may want to check out the Common Crawl dataset: http://commoncrawl.org/
It's hosted on Amazon s3 as HDFS with a 'pay as you access' model.