3

When i'm try to synchronize my s3 bucket with openFDA S3 bucket ( aws s3 sync s3://my-bucket s3://download.open.fda.gov/), error occurce:

"A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied"

Can you tell me, what am I doing wrong?

3

It took quite a while to figure this out myself, so I thought I'd share!

The openFDA bucket is Requestor Pays, so you have to enable yourself -- in your amazon account -- to pay for requests to that bucket. To do so, create a policy document on a group that you are in.

Steps:

1. Create an IAM group called openFDA.

2. Create a policy called openFDABucketAccess

3. Attach the policy to the group.

4. Add your id to the group.

5. Test it using policy simulator. Try to simulate a getObject request on arn:aws:s3:::download.open.fda.gov. It should work!

In the following policy, the only thing you should change is 'my-open-fda-bucket' to your real bucket name that you plan to sync to.

{ "Version": "2016-03-14", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:Get*", "s3:List*" ], "Resource": "arn:aws:s3:::my-open-fda-bucket" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:Get*", "s3:List*" ], "Resource": "arn:aws:s3:::download.open.fda.gov" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListObject", "s3:Get*", "s3:List*" ], "Resource": "arn:aws:s3:::my-open-fda-bucket/*" } ] }

Now that you have the permission (and you will pay to get files from that bucket), you want to sync by using the following aws s3 command:

aws s3 sync s3://download.open.fda.gov s3://my-open-fda-bucket --source-region us-east-1 --region [my-open-fda-bucket-region]

The last example on this page: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html shows you how to sync files between two buckets in different regions. --region is the target region, while --source-region is the source region.

  • Thank you so much for sharing! I will create an issue on the open.fda.gov github repo to have this added to the documentation. – Hans Nelsen Mar 15 '16 at 17:23
  • The only word of caution I would add to your s3 command is that all of the archives for the downloads and all of the Structured Product Labels (SPL) are in that same bucket. If you are only interested in the current API downloads, then I would make three separate requests by appending the following three names to your command: food, drug and device. For example: aws s3 sync s3://download.open.fda.gov/device s3://my-open-fda-bucket/device --source-region us-east-1 --region [my-open-fda-bucket-region] – Hans Nelsen Mar 15 '16 at 17:30
  • Hi Hans - you're welcome, glad to help. Hey thanks! I wish I'd thought of that - a bucket for each type - doh! Is there any way to make it go faster? Mine seems to have stopped for some reason and I don't know if I got everything. I believe i have about 213,000 files and 208GB. How much data is in that bucket? ALSO Is there a map of the data in the bucket? Seems it goes /2015-12-22/drug/event (or label or enforcement) as well as /drug/event/2004q1/drug-event-0001-0001.json.zip – Darby Apr 4 '16 at 5:58
  • @Taras - could you please mark my answer as correct? :-) – Darby Apr 4 '16 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.