I'm trying to export all data from CKAN instances for a big data project. How can I do this?
Did you have a look at CKAN's extensive API documentation? To get you started, look at the answer to related question How do I get a full list of datasets available on Data.Gov using the CKAN API? and its example of using
Once you have a list of dataset IDs, you can get their metadata using the
package_show API function.
url field sometimes contains directly downloadable files, sometimes they seem link to data provider pages which might need some individual handling (i.e. coding). But only restricting batch download to recognised filetypes (e.g.
*.csv) should get you pretty far already.