I'm looking for a list of benign urls. I've seen that others have used the Yahoo URL Random Generator (random.yahoo.com/bin/ryl), but that link doesn't seem to work anymore. The other thing that I thought might work was seeding a crawler with the Alexa list.
3
-
what is a benign URL? a kind, generous URL?– albertCommented Jun 28, 2016 at 15:42
-
2I just meant not malicious. Not malware, phishing, etc.– MASCommented Jun 28, 2016 at 17:30
-
If you can find a way to guarantee non-maliciousness, I think there's a patent or two waiting to be written.– pipeCommented Jul 3, 2016 at 23:36
Add a comment
|
1 Answer
You can use the Common Crawl data, which is loosely related to the Internet Archive and Wayback Machine
the Common Crawl Corpus encompasses over two petabytes of web crawl data collected over eight years and ongoing. As the largest, most comprehensive, open repository of web crawl data on the cloud, we contribute to the thriving open data commons that drives innovation, research, and education.
In particular, the URL index they use for the crawl
- https://github.com/trivio/common_crawl_index
- Direct download (218 GB, http://commoncrawl.org/2013/01/common-crawl-url-index/)
-
-
To be sure, I'd ask them, and be clear about how you define "benign" (porn yes, phishing no)– philshemCommented Jun 30, 2016 at 13:42