17

Data.gov provides ~400,000 datasets from 180 federal agencies. However, I know you are interested in more data and more specific data than what is already included. What additional topics, formats, or types of data are you interested in having published from the government?

(You can always request specific datasets at https://explore.data.gov/nominate ).

  • 3
    This is largely a subjective question without a "correct" answer. – Clay Johnson May 13 '13 at 13:18
  • @ClayJohnson: Surely there are holes in the federal government's offerings that are worth pointing out, especially if together they might form identifiable categories of closed information? – Roger_S May 14 '13 at 3:38
  • 3
    I understand there's not a "correct" answer, but did feel that the members of this community would be the right people to help answer the question from their own perspectives. The disclaimer I should make is that I serve as the Evangelist for Data.gov and would work to help close the data gaps identified here and prompt agencies to offer data that is requested. – Jeanne Holm May 14 '13 at 21:16
  • 4
    The notion that there must be a "correct" answer is a weakness of the stack overflow format. – jfenton May 15 '13 at 4:54
  • 1
    I'll take that as a serious question. Here is the official agency information about investigation into this topic: nasa.gov/centers/glenn/technology/warp/warpfaq_prt.htm Note that unidentified flying objects (UFOs) are "unidentified" and that upon analysis many of them have been identified as atmospheric phenomena, aircraft, or other human made materials. NASA notes, "no branch of the United States Government is currently involved with or responsible for investigations into the possibility of alien life on other planets or for investigating Unidentified Flying Objects (UFO's)." – Jeanne Holm May 15 '13 at 15:12
9

Some of the information you are looking for can be found by looking at the tags portion of the web site you can see how people are tagging their questions. Of the More popular tags the one I think you would most interested in is API (at the time of this writing approximately 1 in 7 questions are tagged with API). I think the most important thing in an API is accessibility, and from my experience accessibility can be broken down into a couple different areas.

  1. How much effort does it take to get the data into a native data structure within my program?

    • Are there third party libraries like python-twitter for Twitter?
    • Is the data in a standard format like JSON or XML? (coming from a python background JSON is preferable to XML)
  2. How well documented is the API and how easy is it to find and navigate the documentation?

    • Does the documentation have broken links, was it's last update in usenet post from 1995?

    • Are there code examples in the documentation that I can simply copy and paste into my editor and get instant results? (The less time I have to spend figuring out how something works the faster I can develop... Duh :-)

    • Is deprecated functionality annotated? Are alternative methods provided for deprecated functionality?

  3. Is the data provided by the API complete and/or meaningful?

    • Are there specific error messages? (i.e. You entered an incorrect date format is a much better error message than Your input is invalid)
    • I run into a lot of situations where an API has a list function of some kind and the list function will only return a portion of the expected results. (i.e. if query example.com/api.php?list=letters and get back anything other than ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'] there's a problem. If there is some reason for not sending an API user all 10 million baby names from your baby name database explain that to them and offer an alternative way to get the data.

    • Are there explanations for all the properties found in the data? (i.e. an API response gives me a something like {'license':None, 'data':[1,2,3]} does that mean that there is no license associated with the data or does it mean that the API providers don't have a license on record but one may exist. Ambiguity is bad.)

To sum this up I don't think there is a problem with the amount of data or variety of data but rather with accessibility.

1

A VERY SPECIFIC DATASET: Hi @Jeanne, I have been scouring the internet in search of a 65-Million-Year temperature history (isotopic record) dataset. I put in a request at data.gov as suggested. I created this question on opendata.stackexchange which has led me in the right direction, and I am still pursuing the leads generated there, but I have not come up with anything solid yet...

4

I'd love to see more city/state data in specific categories. Each may or may not be rolled up to the federal level currently (I'm not totally sure).

So here's a list, and you can let us know if it's possible to get address-specific data for each one, or if the feds will always get aggregated, and non-address specific data, for these data sets.

  • Individual crime reports: seems to stop at the city level, then goes to UCR categories by Zip at the state level.

  • Traffic accidents: seems to stop at the state level.

  • Sex offender registry: stops at the state level, seems odd to me there is no national registry.

  • Property values (taxed value) of parcels.

  • Restaurant health inspection scores. Currently city level, though Yelp has helped create a national standard.

  • Political contributions - individual and company contributions to all political campaigns, down the street address level. I think the feds have this data, but I'm not sure where to get it in bulk.

  • Foreclosures

  • Recent home sales. I know this is tracked locally, but not much higher.

  • Building permits.

  • 311 Calls for service.

  • Special Business licenses, eg, alcohol, advertising, pornography, etc.

I realize many of these datasets might have privacy issues attached to them, but they are all already released at least at the city level (and sensitive data is already scrubbed/fuzzed). I don't see an issue with aggregating them at the state/fed level, unless you are concerned with the federal government collecting too much information about its citizens, which is another possible issue.

6

Maybe there are privacy implications I haven't thought of, but it would be great if the IRS would release a database of EIN, legal name, and jurisdiction of record for all US businesses that pay taxes. (I don't mean tax $ amounts, just identifying info)

  • 1
    I'll look into that. In the meantime, there are a few resources. The Defense Logistics Agency runs a Business Identification Numbers Cross-referencing (BINCS) System dlis.dla.mil/bincs/begin_search.aspx that may provide some information, but doesn't appear to be downloadable as a dataset. There is also a central contractor interactive dataset at explore.data.gov/Information-and-Communications/… that lets you explore more information. – Jeanne Holm May 16 '13 at 17:05
  • For publicly traded companies there should be no barrier to providing the tax $ as well as other information (revenue, profit, whatever else is publically available), should there? It would be nice to get that in one API call instead of rooting around for it. Or does that already exist, @JeanneHolm? – cori May 21 '13 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.