How can I download AWS Public Datasets?

Question

https://aws.amazon.com/datasets/ contain many datasets. How can I download an AWS public dataset?

Answer 1

I'll take the example of https://aws.amazon.com/datasets/enron-email-data/.

1. Note the snapshot ID of the data set you plan to download. As indicated on https://aws.amazon.com/datasets/enron-email-data/., Enron-email-data's snapshot ID is snap-d203feb5.

2. Locate the AWS region in which the snapshot is located. To do, search for the snapshot ID in the list of existing snapshots, e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId . If you don't find the snapshot, you have to change of AWS region:

   

   In the example of Enron-email-data, it is located in US East (N. Virginia) as well as in US Oregon. (In case you wondered: How can I know in which US region a snapshot ID is located? -> short answer: you have to try all regions yourselves by trying each region in the menu as shown in the screenshot above.)

3. Create an EC2 instance in the same region as the snapshot, i.e. in our example Virginia a.k.a. US-East-1. To do go to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Home and:

   

4. Once the EC2 instance is launched and running, look at its availability zone on https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceId (change the AWS region to the region where you launched the EC2 instance), which in our case us-east-1c:

   

5. Go back to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId and right click on the snapshot and create a volume:

   

   Make sure you select the same availability zone as the one where your EC2 instant is running, which in our case us-east-1c:

   

6. In your volume list (e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Volumes:sort=desc:createTime if you're in US East N. Virginia), right-click on the version you have created, and select "Attach Volume".

   

   It will ask you to which EC2 instance the volume should be attached. Select your EC2 instance.

   

7. SSH into your EC2 instance, and mount the volume as follows:

   lsblk # <-- it'll give you the name  of the volume, in our case it is xvdf
   sudo mkdir /enron
   sudo mount xvdf /enron
   

Done!

---

Below are some tricks to transfer the corpus from the EC2 to your computer.

To transfer the corpus from the EC2 to your computer, assuming that AWSvirginia.pem is the private key of the EC2 instance, and that the 184.72.123.192 is the public IP of the EC2 instance:

chmod 400 AWSvirginia.pem 
ssh -i AWSvirginia.pem ec2-user@184.72.123.192
rsync -Pav -e "ssh -i AWSvirginia.pem"  ec2-user@184.72.123.192:/enron/ .

After the transfer is completed, you can do additional checks as follows (even though rsync already checks for file corruption during the transfer):

To get the shasum hash for each file in a directory and its subdirectories (and recursively):

find . -type f -print0 | sort -z | xargs -0 sha1sum > checksums.txt

To check if the files match the shasum hashes:

sha1sum -c checksums.txt 

If there is any issues, it will display at the end of the checksum:

sha1sum: WARNING: 1 computed checksum did NOT match

To check every zip-file in every subfolder:

find . -type f -iname '*.zip' -exec unzip -tq {} \;

---

Regarding the AWS costs, there 3 types of you should be aware of:

• Price of running an EC2 instance
• Price of transferring data from EC2 instance to outside AWS
• Price of having a volume