Catalog of all websites on the entire internet

Question

Needs

For a project, I am trying to run analysis on a representative sample of all websites on the entire internet. I'm trying to do things like measure market penetration of certain web technologies, hosting providers, etc.

domainsproject.org

I stumbled across this dataset - is it legit? Has anyone used this data before? It's based on crawlers, so it'll be biased towards websites that actually get traffic, which for my purposes is totally fine. The git repo just looks a bit stale - the only people commenting are the maintainers. Before I do much work with this, I'd like a bit of validation from the community.

Other options

Does anyone have any other suggestions on how to get a representative sample of all websites on the entire internet?

Answer 1

Regarding git repo: it's not stale, it was updated about a month ago. Still, there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it with mine.

Anyway, here are some relevant posts:

1. Dataset of domain names
2. List of all http sites in the world