2

I'm looking for a list of benign urls. I've seen that others have used the Yahoo URL Random Generator (random.yahoo.com/bin/ryl), but that link doesn't seem to work anymore. The other thing that I thought might work was seeding a crawler with the Alexa list.

  • what is a benign URL? a kind, generous URL? – albert Jun 28 '16 at 15:42
  • 2
    I just meant not malicious. Not malware, phishing, etc. – MAS Jun 28 '16 at 17:30
  • If you can find a way to guarantee non-maliciousness, I think there's a patent or two waiting to be written. – pipe Jul 3 '16 at 23:36
2

You can use the Common Crawl data, which is loosely related to the Internet Archive and Wayback Machine

the Common Crawl Corpus encompasses over two petabytes of web crawl data collected over eight years and ongoing. As the largest, most comprehensive, open repository of web crawl data on the cloud, we contribute to the thriving open data commons that drives innovation, research, and education.

(wiki page)


In particular, the URL index they use for the crawl

  • Can I assume that all of the urls are non-malicious? – MAS Jun 30 '16 at 12:28
  • To be sure, I'd ask them, and be clear about how you define "benign" (porn yes, phishing no) – philshem Jun 30 '16 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.