https://aws.amazon.com/datasets/ contain many datasets. How can I download an AWS public dataset?

  • @StanislavKralin: could you please explain your downvote? – Franck Dernoncourt Apr 15 at 18:34
  • Reason for downvote: Public and Open data are essentially different ... and you did show that in your answer: Public accessible with AWS infrastructure. In order for the data to be open than need to be infrastructure less, get the API or simple download into my machine and start using ... AWS Public datasets are "tied" to their infrastructure (due the size). – n1tk Apr 16 at 5:10
  • @sb0709 "AWS Public datasets are "tied" to their infrastructure (due the size)" -> What do you mean tied? Some AWS public datasets are small, e.g. aws.amazon.com/datasets/freebase-simple-topic-dump/… is 5 GB. – Franck Dernoncourt Apr 16 at 5:15
  • @sb0709 "simple download" -> If you know how to use AWS, it takes less than one minute to initiate the download of an AWS Public dataset. – Franck Dernoncourt Apr 16 at 5:16
  • here is the amazom docs: docs.aws.amazon.com/AWSEC2/latest/UserGuide/… – n1tk Apr 16 at 5:17
up vote 2 down vote accepted

I'll take the example of https://aws.amazon.com/datasets/enron-email-data/.

  1. Note the snapshot ID of the data set you plan to download. As indicated on https://aws.amazon.com/datasets/enron-email-data/., Enron-email-data's snapshot ID is snap-d203feb5.
  2. Locate the AWS region in which the snapshot is located. To do, search for the snapshot ID in the list of existing snapshots, e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId . If you don't find the snapshot, you have to change of AWS region:

    enter image description here

    In the example of Enron-email-data, it is located in US East (N. Virginia) as well as in US Oregon. (In case you wondered: How can I know in which US region a snapshot ID is located? -> short answer: you have to try all regions yourselves by trying each region in the menu as shown in the screenshot above.)

  3. Create an EC2 instance in the same region as the snapshot, i.e. in our example Virginia a.k.a. US-East-1. To do go to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Home and:

    enter image description here

  4. Once the EC2 instance is launched and running, look at its availability zone on https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceId (change the AWS region to the region where you launched the EC2 instance), which in our case us-east-1c:

    enter image description here

  5. Go back to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId and right click on the snapshot and create a volume:

    enter image description here

    Make sure you select the same availability zone as the one where your EC2 instant is running, which in our case us-east-1c:

    enter image description here

  6. In your volume list (e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Volumes:sort=desc:createTime if you're in US East N. Virginia), right-click on the version you have created, and select "Attach Volume".

    enter image description here

    It will ask you to which EC2 instance the volume should be attached. Select your EC2 instance.

    enter image description here

  7. SSH into your EC2 instance, and mount the volume as follows:

    lsblk # <-- it'll give you the name  of the volume, in our case it is xvdf
    sudo mkdir /enron
    sudo mount xvdf /enron
    

Done!


Below are some tricks to transfer the corpus from the EC2 to your computer.

To transfer the corpus from the EC2 to your computer, assuming that AWSvirginia.pem is the private key of the EC2 instance, and that the 184.72.123.192 is the public IP of the EC2 instance:

chmod 400 AWSvirginia.pem 
ssh -i AWSvirginia.pem ec2-user@184.72.123.192
rsync -Pav -e "ssh -i AWSvirginia.pem"  ec2-user@184.72.123.192:/enron/ .

After the transfer is completed, you can do additional checks as follows (even though rsync already checks for file corruption during the transfer):

To get the shasum hash for each file in a directory and its subdirectories (and recursively):

find . -type f -print0 | sort -z | xargs -0 sha1sum > checksums.txt

To check if the files match the shasum hashes:

sha1sum -c checksums.txt 

If there is any issues, it will display at the end of the checksum:

sha1sum: WARNING: 1 computed checksum did NOT match

To check every zip-file in every subfolder:

find . -type f -iname '*.zip' -exec unzip -tq {} \;

Regarding the AWS costs, there 3 types of you should be aware of:

  • Price of running an EC2 instance
  • Price of transferring data from EC2 instance to outside AWS
  • Price of having a volume
  • i appreciate all of this, but a) open data doesn't require sign ups and b) has zero implicit associated costs. also, the title is public datasets, but you point to datasets. yes, those are public, but amazon has a different url/data for the public datasets. – albert Apr 14 at 3:09
  • @albert has this been decided somewhere on meta? – Franck Dernoncourt Apr 14 at 4:25
  • not that i'm aware of. maybe bring it up there? – albert Apr 14 at 16:50
  • @albert well, regarding a) this website does already discuss data that require sign ups (e.g., MIMIC dataset); regarding b) I view AWS as implicit costs (you don't pay for the corpus; you pay for the host infrastructure; and anyway for small corpora, AWS is free). – Franck Dernoncourt Apr 14 at 17:45
  • points are noted, but don't change the facts around open data. there are ways to access aws datasets and aws public datasets without using their infrastructure and tools. those methods would be appropriate here. – albert Apr 15 at 1:47

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.