1

All gov sites from national level down to town/village level. And their official social media profile handles. Is there such a dataset available?

  • Global? US? federal? – philshem Nov 5 at 16:27
  • US, but global would be great too! – user21861 Nov 5 at 21:35
  • This is the opposite of helpful, but: czds.icann.org will let you download entire zone files for certain TLDs, but, sadly, .gov is not one of them. You might look at commoncrawl.org indexed URLs but this won't necessarily be complete. – Barry Carter Nov 8 at 2:00
  • If anything, .gov should be freely available from icaan! I'm surprised it isn't, thank you for the link – user21861 Nov 8 at 18:49
1

The DIY way would be to to compile the lists of websites with a bit of internet searching:

https://www.nclc.org/for-consumers/us-government-websites.html

https://www.usa.gov/federal-agencies

and then from each page source scrape social media accounts

<div class="usa-footer-secondary_section usa-footer-big-secondary-section">
   <div class="usa-grid">
      <div class="usa-footer-contact-links">
         <ul class="usa-unstyled-list noborder">
            <li> <a href="https://www.facebook.com/USAgov"><img  class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Facebook.png" alt="Facebook USAGov"/></a> </li>
            <li> <a href="https://twitter.com/USAgov"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Twitter.png" alt="Twitter USAGov"/></a> </li>
            <li> <a href="https://www.youtube.com/usagov1"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Youtube.png" alt="YouTube USAGov"></a> </li>
            <li> <a href="https://www.instagram.com/usagov/"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Instragram.png" alt="Instagram USAGov"></a> </li>
            <li> <a href="https://www.snapchat.com/add/usagov"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Snapchat.png" alt="Snapchat USAGov"/></a> </li>
            <li> <a href="https://www.pinterest.com/usagov/"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_Pinterest.png" alt="Pinterest USAGov"></a> </li>
            <li> <a href="https://www.usa.gov/rss"><img class="sm-image" src="/sites/all/themes/usa/images/Icon_Connect_RSS.png" alt="RSS USAGov"/></a> </li>
         </ul>
      </div>
   </div>
</div>
1

This is not an answer.

I got bored, so, using commoncrawl.org data dated 26 Sep 2018 (but not all collected at the same time, of course), I managed to extract a list of 30,815 .gov domains in the "gov-domains.txt" file at http://test.barrycarter.info/20191107/

A larger list of all 69,051,164 known to commoncrawl.org on the same date is at the same URL in the file "alldomains.txt.bz2" (bzip2 compressed). Please use right-click "save link as", since clicking on the link directly may not work.

The second list is in a slightly different format where the subdomains are reversed and separated by commas. For example:

gov,wa,houserepublicans,andrewbarkis

means andrewbarkis.houserepublicans.wa.gov

According to https://blog.verisign.com/domain-names/verisign-q1-2019-domain-name-industry-brief-internet-grows-to-351-8-million-domain-name-registrations-in-the-first-quarter-of-2019/ there are over 350 million registered domain names, so this list is nowhere near complete. Additionally, registered domain names are domains like "google.com", and don't include subdomains like "search.google.com". Since my list above does include subdomains, it is even more incomplete.

I assumed by "gov" you meant ".gov" names, but the second list can you help you find other government domains as well. In particular, domains ending in state.xx.us where xx is a state postal code abbreviation are official US state government domains (https://en.wikipedia.org/wiki/.us#States_and_territories), though many now redirect to friendlier names. For example, www.state.nm.us now redirects to newmexico.gov.

Of course, some localities opt not to use .gov domains at all, such as: https://www.corrales-nm.org/

Finally, non-US government domains often end in go.xx or gov.xx, where xx is a country code.

  • Wow, this is awesome! I upvoted, but I don't have enough points (or whatever) so it doesn't show up here. Thank you for pointing me to commoncrawl – user21861 Nov 8 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.