Approach #1 is doable, but not necessary. We'll go with approach #2 here: Download metadata provided by Socrata.
The Open Data Network has an official API that you can use to query the various entities and datasets. You should be able to use that to find all datasets in the network (it looks like that might be doable through a single query, without having to script your way past pagination). Be mindful of usage thresholds so you don't get throttled, and be sure to create an API key, rather than abuse the one from the demo page. Each entry has a few fields that will be useful for fetching metadata and other related tasks:
fxf
is an ID associated with the dataset (there are multiple...their difference is a little murky). For clarity and consistent naming I'll call it id_no
from here on.
domain
is the entity domain for this dataset (Socrata's client and the owner of the data)
domain_url
is the root url for the data portal hosting the dataset
dataset_url
is the human readable "about the dataset" page
dev_docs_url
provides more information (human readable) about the dataset for API users
The metadata itself is available in a few places. If you just want the row count, one of the easiest ways to get it is via a request to <domain_url>/api/id/<id_no>.json?$select=count(*)+AS+count
. This uses Socrata's query language to query the dataset directly. This is also what the dev_docs_url
page uses under the hood to display that info.
For more extensive metadata, which includes field information, null count by column, update frequency and times, etc., you can query <domain_url>/api/views/<id_no>.json
. I don't think number of columns is explicitly included but the columns
array's length should give you that value.