3

I've tried inurl:http but it takes forever to get even a bunch of sites right and I have to think of new keywords everytime to get the sites.

Is there kind of a directory I could use to get the address of all the sites on the public web?

  • Also HTTPS I guess? – Nicolas Raoul Jun 7 '19 at 3:59
  • 1
    what is inurl:http – albert Jun 7 '19 at 5:03
  • 1
    you probably do not want to run this: curl -IL http://{0..255}.{0..255}.{0..255}.{0..255} 😂 – philshem Jun 7 '19 at 13:15
  • 1
    I think inurl:http is an attempted hack on google advance searching: ahrefs.com/blog/google-advanced-search-operators, but I'm pretty sure google doesn't consider the application protocol to be meaningful when indexing the sites. – philshem Jun 7 '19 at 13:40
  • commoncrawl.org would probably be your best bet. – user3856 Jun 24 '19 at 17:47
2

There are an estimated 1.6 billion public websites in the world, with 200 million being active. But no one really knows, because a website can be just an IP address with no domain, or an .onion link, or temporary or short-lived, or weather-dependent, ...


One way would be to use known DNS records (domain names) from the 2013 DNS Census.

It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB.

They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains


A more modest list would be, for example, the Alexa 1 million list:

Scripts for scanning the Alexa top 1 million sites and providing generic statistics about them.

Direct link: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip


Or, like my comment, loop over IPv4 addresses... and record if each IP is a valid http/https server.

Here's an estimate about how big your for loop will get:

According to Reserved IP addresses there are 588,514,304 reserved addresses and since there are 4,294,967,296 (2^32) IPv4 addressess in total, there are 3,706,452,992 public addresses.

| improve this answer | |
  • Does your number include port numbers – nmtoken Jul 11 '19 at 7:03
  • @nmtoken nope! but if you have more cycles you can always add all the ports that aren't usually associated with http(s) to see if http(s) is running there ;) – philshem Jul 11 '19 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.