3

I've tried inurl:http but it takes forever to get even a bunch of sites right and I have to think of new keywords everytime to get the sites.

Is there kind of a directory I could use to get the address of all the sites on the public web?

2

There are an estimated 1.6 billion public websites in the world, with 200 million being active. But no one really knows, because a website can be just an IP address with no domain, or an .onion link, or temporary or short-lived, or weather-dependent, ...


One way would be to use known DNS records (domain names) from the 2013 DNS Census.

It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB.

They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains


A more modest list would be, for example, the Alexa 1 million list:

Scripts for scanning the Alexa top 1 million sites and providing generic statistics about them.

Direct link: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip


Or, like my comment, loop over IPv4 addresses... and record if each IP is a valid http/https server.

Here's an estimate about how big your for loop will get:

According to Reserved IP addresses there are 588,514,304 reserved addresses and since there are 4,294,967,296 (2^32) IPv4 addressess in total, there are 3,706,452,992 public addresses.

  • Does your number include port numbers – nmtoken Jul 11 at 7:03
  • @nmtoken nope! but if you have more cycles you can always add all the ports that aren't usually associated with http(s) to see if http(s) is running there ;) – philshem Jul 11 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.