For a project, I am trying to run analysis on a representative sample of all websites on the entire internet. I'm trying to do things like measure market penetration of certain web technologies, hosting providers, etc.
I stumbled across this dataset - is it legit? Has anyone used this data before? It's based on crawlers, so it'll be biased towards websites that actually get traffic, which for my purposes is totally fine. The git repo just looks a bit stale - the only people commenting are the maintainers. Before I do much work with this, I'd like a bit of validation from the community.
Does anyone have any other suggestions on how to get a representative sample of all websites on the entire internet?