Get metadata from all datasets on Socrata

Question

Socrata.org gives a possibility to browse all datasets that are published by their customers with their API via Open Data Network.

I'd like to collect metadata such as:

• Dataset size
• Number of rows
• Number of columns
• Number of tables (if applicable)

for as many datasets as possible (possibly all of them).

I see two possible approaches here:

1. Download whole dataset and calculate metadata myself.
   How to automate download of all available datasets?
2. Download only metadata that has already been calculated by Socrata and is published in dataset description.
   What is the API that I should use? Is it possible at all?

Answer 1

Approach #1 is doable, but not necessary. We'll go with approach #2 here: Download metadata provided by Socrata.

The Open Data Network has an official API that you can use to query the various entities and datasets. You should be able to use that to find all datasets in the network (it looks like that might be doable through a single query, without having to script your way past pagination). Be mindful of usage thresholds so you don't get throttled, and be sure to create an API key, rather than abuse the one from the demo page. Each entry has a few fields that will be useful for fetching metadata and other related tasks:

• fxf is an ID associated with the dataset (there are multiple...their difference is a little murky). For clarity and consistent naming I'll call it id_no from here on.
• domain is the entity domain for this dataset (Socrata's client and the owner of the data)
• domain_url is the root url for the data portal hosting the dataset
• dataset_url is the human readable "about the dataset" page
• dev_docs_url provides more information (human readable) about the dataset for API users

The metadata itself is available in a few places. If you just want the row count, one of the easiest ways to get it is via a request to <domain_url>/api/id/<id_no>.json?$select=count(*)+AS+count. This uses Socrata's query language to query the dataset directly. This is also what the dev_docs_url page uses under the hood to display that info.

For more extensive metadata, which includes field information, null count by column, update frequency and times, etc., you can query <domain_url>/api/views/<id_no>.json. I don't think number of columns is explicitly included but the columns array's length should give you that value.