For a project, I am trying to run analysis on a representative sample of all websites on the entire internet. I'm trying to do things like measure market penetration of certain web technologies, hosting providers, etc.


I stumbled across this dataset - is it legit? Has anyone used this data before? It's based on crawlers, so it'll be biased towards websites that actually get traffic, which for my purposes is totally fine. The git repo just looks a bit stale - the only people commenting are the maintainers. Before I do much work with this, I'd like a bit of validation from the community.

Other options

Does anyone have any other suggestions on how to get a representative sample of all websites on the entire internet?


1 Answer 1


Regarding git repo: it's not stale, it was updated about a month ago. Still, there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it with mine.

Anyway, here are some relevant posts:

  1. Dataset of domain names
  2. List of all http sites in the world
  • Hi Bodan - thanks for your effort! It's a great resource.
    – philshem
    Nov 26, 2020 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.