3

I've tried inurl:http but it takes forever to get even a bunch of sites right and I have to think of new keywords everytime to get the sites.

Is there kind of a directory I could use to get the address of all the sites on the public web?

Improve this question
5
  • Also HTTPS I guess?
    – Nicolas Raoul ♦
    Commented Jun 7, 2019 at 3:59
  • 1
    what is inurl:http
    – albert
    Commented Jun 7, 2019 at 5:03
  • 1
    you probably do not want to run this: curl -IL http://{0..255}.{0..255}.{0..255}.{0..255} 😂
    – philshem
    Commented Jun 7, 2019 at 13:15
  • 1
    I think inurl:http is an attempted hack on google advance searching: ahrefs.com/blog/google-advanced-search-operators, but I'm pretty sure google doesn't consider the application protocol to be meaningful when indexing the sites.
    – philshem
    Commented Jun 7, 2019 at 13:40
  • commoncrawl.org would probably be your best bet.
    – user3856
    Commented Jun 24, 2019 at 17:47

1 Answer 1

Reset to default
3

There are an estimated 1.6 billion public websites in the world, with 200 million being active. But no one really knows, because a website can be just an IP address with no domain, or an .onion link, or temporary or short-lived, or weather-dependent, ...

One way would be to use known DNS records (domain names) from the 2013 DNS Census.

It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB.

They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains

A more modest list would be, for example, the Alexa 1 million list:

Scripts for scanning the Alexa top 1 million sites and providing generic statistics about them.

Direct link: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

Or, like my comment, loop over IPv4 addresses... and record if each IP is a valid http/https server.

Here's an estimate about how big your for loop will get:

According to Reserved IP addresses there are 588,514,304 reserved addresses and since there are 4,294,967,296 (2^32) IPv4 addressess in total, there are 3,706,452,992 public addresses.

Improve this answer
2
  • Does your number include port numbers
    – nmtoken
    Commented Jul 11, 2019 at 7:03
  • @nmtoken nope! but if you have more cycles you can always add all the ports that aren't usually associated with http(s) to see if http(s) is running there ;)
    – philshem
    Commented Jul 11, 2019 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.