12

There are many web resources to find domain names (whois.com), and using the WHOIS protocol there are some APIs. Some examples are the unix command line tool jwhois and the python library pywhois. These tools return the full WHOIS record, which includes personal information like name, phone number, and address. Because of the personal information, these APIs either cost money (whoisxmlapi.com) or have IP-address quotas and rate limiting.

I'm not looking for personal information, but I would like to collect non-personal domain data in bulk.

Is there an open dataset of registered domains? (Even old, partial, or geographically limited data would be of interest.)

Improve this question

6 Answers 6

Reset to default
3

If this still makes sense, there's a dataset that I'm maintaining:

https://github.com/tb0hdan/domains

TLD kinds: 1522

Country TLDs: 245

Generic TLDs: 1277

Total domains in dataset: 1,789,946,688

Improve this answer
0
14

I think this is what you are looking for. It is a DNS registration dataset snapshot taken in 2013. Compressed - it is ~15GB and uncompressed 157GB.

http://dnscensus2013.neocities.org/

They claim it contains: Dataset containing 2,676,380,336 DNS records and 106,928,034 domains

Improve this answer
4
  • 1
    Wow. I'm going to need more hardware!
    – philshem
    May 28, 2014 at 18:54
  • 2
    If anyone downloads the file, can you send or share a small piece, like the top few lines (so I can see if it contains the data I need)?
    – philshem
    Jun 27, 2014 at 9:42
  • The torrent file is 15gb. It's a folder with compressed files for each second-level domains (i.e. .com). Thanks for the great data set!
    – philshem
    Aug 14, 2014 at 21:14
  • @philshem Does it have historical records of whois data changes?
    – Anton Tarasenko
    May 13, 2015 at 16:17
8

This isn't a complete dataset, but you will find all .gov.uk domain names with the following info:

  • Domain Name
  • Representing
  • Owner
  • Status
  • Next Renewal Date

Link: http://data.gov.uk/dataset/list-of-gov-uk-domain-names

Improve this answer
1
  • Very cool! It's a good idea to look for gov domains.
    – philshem
    Apr 25, 2014 at 16:47
7

Here you can find the full list of .fr domain names:

http://opendata.afnic.fr/en/products-and-services/services/opendata-en.html

And here the list of .se and .nu domain names:

https://zonedata.iis.se/

Improve this answer
3

To expand on the answer of Anastasios Ventouris, I found an official list for US .gov domains.

Federal Executive Agency Internet Domains as of 03272014

There are also many filter, view, and export options (i.e. JSON). I don't know how the URL will be updated since it includes date information.

Improve this answer
2

Another source of domain names is https://www.spamhaus.org/ a non-profit organization that tracks spam related activities.

They have a Domain Block List (DBL) that includes spam domains: https://www.spamhaus.org/dbl/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.