I'm looking for a list of benign urls. I've seen that others have used the Yahoo URL Random Generator (random.yahoo.com/bin/ryl), but that link doesn't seem to work anymore. The other thing that I thought might work was seeding a crawler with the Alexa list.
the Common Crawl Corpus encompasses over two petabytes of web crawl data collected over eight years and ongoing. As the largest, most comprehensive, open repository of web crawl data on the cloud, we contribute to the thriving open data commons that drives innovation, research, and education.
In particular, the URL index they use for the crawl
- Direct download (218 GB, http://commoncrawl.org/2013/01/common-crawl-url-index/)