10

There are many web resources to find domain names (whois.com), and using the WHOIS protocol there are some APIs. Some examples are the unix command line tool jwhois and the python library pywhois. These tools return the full WHOIS record, which includes personal information like name, phone number, and address. Because of the personal information, these APIs either cost money (whoisxmlapi.com) or have IP-address quotas and rate limiting.

I'm not looking for personal information, but I would like to collect non-personal domain data in bulk.

Is there an open dataset of registered domains? (Even old, partial, or geographically limited data would be of interest.)

12

I think this is what you are looking for. It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB.

http://dnscensus2013.neocities.org/

They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains

  • 1
    Wow. I'm going to need more hardware! – philshem May 28 '14 at 18:54
  • 2
    If anyone downloads the file, can you send or share a small piece, like the top few lines (so I can see if it contains the data I need)? – philshem Jun 27 '14 at 9:42
  • The torrent file is 15gb. It's a folder with compressed files for each second-level domains (i.e. .com). Thanks for the great data set! – philshem Aug 14 '14 at 21:14
  • @philshem Does it have historical records of whois data changes? – Anton Tarasenko May 13 '15 at 16:17
8

This isn't a complete dataset, but you will find all .gov.uk domain names with the following info:

  • Domain Name
  • Representing
  • Owner
  • Status
  • Next Renewal Date

Link: http://data.gov.uk/dataset/list-of-gov-uk-domain-names

  • Very cool! It's a good idea to look for gov domains. – philshem Apr 25 '14 at 16:47
7

Here you can find the full list of .fr domain names:

http://opendata.afnic.fr/en/products-and-services/services/opendata-en.html

And here the list of .se and .nu domain names:

https://zonedata.iis.se/

3

To expand on the answer of Anastasios Ventouris, I found an official list for US .gov domains.

Federal Executive Agency Internet Domains as of 03272014

There are also many filter, view, and export options (i.e. JSON). I don't know how the URL will be updated since it includes date information.

2

Another source of domain names is https://www.spamhaus.org/ a non-profit organization that tracks spam related activities.

They have a Domain Block List (DBL) that includes spam domains: https://www.spamhaus.org/dbl/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.