https://aws.amazon.com/datasets/ contain many datasets. How can I download an AWS public dataset?
-
@StanislavKralin: could you please explain your downvote? – Franck Dernoncourt Apr 15 '18 at 18:34
-
Reason for downvote: Public and Open data are essentially different ... and you did show that in your answer: Public accessible with AWS infrastructure. In order for the data to be open than need to be infrastructure less, get the API or simple download into my machine and start using ... AWS Public datasets are "tied" to their infrastructure (due the size). – n1tk Apr 16 '18 at 5:10
-
@sb0709 "AWS Public datasets are "tied" to their infrastructure (due the size)" -> What do you mean tied? Some AWS public datasets are small, e.g. aws.amazon.com/datasets/freebase-simple-topic-dump/… is 5 GB. – Franck Dernoncourt Apr 16 '18 at 5:15
-
@sb0709 "simple download" -> If you know how to use AWS, it takes less than one minute to initiate the download of an AWS Public dataset. – Franck Dernoncourt Apr 16 '18 at 5:16
-
here is the amazom docs: docs.aws.amazon.com/AWSEC2/latest/UserGuide/… – n1tk Apr 16 '18 at 5:17
I'll take the example of https://aws.amazon.com/datasets/enron-email-data/.
- Note the snapshot ID of the data set you plan to download. As indicated on https://aws.amazon.com/datasets/enron-email-data/., Enron-email-data's snapshot ID is
snap-d203feb5
. Locate the AWS region in which the snapshot is located. To do, search for the snapshot ID in the list of existing snapshots, e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId . If you don't find the snapshot, you have to change of AWS region:
In the example of Enron-email-data, it is located in US East (N. Virginia) as well as in US Oregon. (In case you wondered: How can I know in which US region a snapshot ID is located? -> short answer: you have to try all regions yourselves by trying each region in the menu as shown in the screenshot above.)
Create an EC2 instance in the same region as the snapshot, i.e. in our example Virginia a.k.a. US-East-1. To do go to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Home and:
Once the EC2 instance is launched and running, look at its availability zone on https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceId (change the AWS region to the region where you launched the EC2 instance), which in our case
us-east-1c
:Go back to https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=public;search=snap-d203feb5;sort=snapshotId and right click on the snapshot and create a volume:
Make sure you select the same availability zone as the one where your EC2 instant is running, which in our case
us-east-1c
:In your volume list (e.g. https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Volumes:sort=desc:createTime if you're in US East N. Virginia), right-click on the version you have created, and select "Attach Volume".
It will ask you to which EC2 instance the volume should be attached. Select your EC2 instance.
SSH into your EC2 instance, and mount the volume as follows:
lsblk # <-- it'll give you the name of the volume, in our case it is xvdf sudo mkdir /enron sudo mount xvdf /enron
Done!
Below are some tricks to transfer the corpus from the EC2 to your computer.
To transfer the corpus from the EC2 to your computer, assuming that AWSvirginia.pem
is the private key of the EC2 instance, and that the 184.72.123.192
is the public IP of the EC2 instance:
chmod 400 AWSvirginia.pem
ssh -i AWSvirginia.pem ec2-user@184.72.123.192
rsync -Pav -e "ssh -i AWSvirginia.pem" ec2-user@184.72.123.192:/enron/ .
After the transfer is completed, you can do additional checks as follows (even though rsync
already checks for file corruption during the transfer):
To get the shasum hash for each file in a directory and its subdirectories (and recursively):
find . -type f -print0 | sort -z | xargs -0 sha1sum > checksums.txt
To check if the files match the shasum hashes:
sha1sum -c checksums.txt
If there is any issues, it will display at the end of the checksum:
sha1sum: WARNING: 1 computed checksum did NOT match
To check every zip-file in every subfolder:
find . -type f -iname '*.zip' -exec unzip -tq {} \;
Regarding the AWS costs, there 3 types of you should be aware of:
- Price of running an EC2 instance
- Price of transferring data from EC2 instance to outside AWS
- Price of having a volume
-
i appreciate all of this, but a) open data doesn't require sign ups and b) has zero implicit associated costs. also, the title is public datasets, but you point to datasets. yes, those are public, but amazon has a different url/data for the public datasets. – albert♦ Apr 14 '18 at 3:09
-
-
-
@albert well, regarding a) this website does already discuss data that require sign ups (e.g., MIMIC dataset); regarding b) I view AWS as implicit costs (you don't pay for the corpus; you pay for the host infrastructure; and anyway for small corpora, AWS is free). – Franck Dernoncourt Apr 14 '18 at 17:45
-
points are noted, but don't change the facts around open data. there are ways to access aws datasets and aws public datasets without using their infrastructure and tools. those methods would be appropriate here. – albert♦ Apr 15 '18 at 1:47