1

I am working on a tool to manage data downloading.

One source of data I would like to make easy to access is the UCI ML dataset repository

A lot (all?) of the data is in directories, for example for the Auto MPG dataset the directory is this one

Now it is possible to just download each file in turn, but that is inelegant. And right now I can't even see a way to do that automatically without scraping HTML.

Does the UCI repo expose a way to download the whole dataset e.g. in a zip-file. Or some nicer automate-able way, like FTP, that would at least let me avoid scraping HTML to get a list of URLs.

Improve this question

2 Answers 2

Reset to default
1

This is not ftp but it is the HTTP directory listing. Sort of the midpoint between HTML and FTP but it might be what you are looking for.

http://mlearn.ics.uci.edu/databases/

Improve this answer
0

This seems like a task for GNU's wget, the open source vacuuming tool used mostly for websites. It's standard on Linux, and can be installed on Mac OS X as well as Windows, or as part of Cygwin. An alternative to wget would be curl.

I would imagine you would create a list of all data folders, and then loop over each folder, running wget for each item in the loop:

wget --recursive --no-parent https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/

See here for more details.

Or, if you can remove the auto-mpg sub-folder and then you'll really vacuum everything.

Programming languages may have ports of wget, for example for python

https://pypi.python.org/pypi/wget

a pure python download utility

Statistical and machine learning programming languages may have pseudo-APIs with the UCI ML Repository.

For R, see for example readMLData:

Functions for reading data sets in different formats for testing machine learning tools are provided. This allows to run a loop over several data sets in their original form, for example if they are downloaded from UCI Machine Learning Repository. The data are not part of the package and have to be downloaded separately.

The download_script.R (source) uses wget for listed data packages

Improve this answer
5
  • Indeed, it looks like vacuuming archive.ics.uci.edu/ml/machine-learning-databases gives all datasets.
    – user4293
    Commented Oct 23, 2017 at 12:36
  • I'm not keen to use wget (or curl), because they don't tend to come with windows and adding it as a dependancy would be too heavy weight. Looking at wget, it included a 1200 line HTML parser for this purpose.I could do something evil by invoking powershell shenanigans in windows to do a --recursive but then it would break on all kinds of different windows setups. Or I could use a Html parse of my own (but that is a dependency). This might be a time when parsing HTML with regex is good enough...
    – Frames Catherine White
    Commented Oct 24, 2017 at 2:11
  • what programming language are you using? using regex to parse html is usually not advised (joke: blog.codinghorror.com/parsing-html-the-cthulhu-way)
    – philshem
    Commented Oct 24, 2017 at 7:15
  • see here for getting standalone wget.exe with no libraries or dlls -- superuser.com/a/245567/442118
    – philshem
    Commented Oct 24, 2017 at 7:22
  • I'm using Julia, and in julia we have a bunch of great HTML parsers. But I am trying for minimal dependencies. Sure generally parsing HTML is bad because HTML is non-regular. But any finite set of documents is regular. And the format of the UCI pages is consistent (it is generated by apache) and don't do anything like include href=".*?" in the body or in comments. I'm not super happy about it though. So maybe just downloading wget.exe from somewhere is best
    – Frames Catherine White
    Commented Oct 25, 2017 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.