I am looking to retrieve every ASIN in Amazon's product database. In setting up a sample scrapy web scraping script - it became clear that crawling is not the efficient solution.
Looking at answers about the Product Advertising API I found a similar question: Amazon ASIN and CategoryAmazon ASIN and Category
If the goal is to retrieve all of the physical products currently listed on Amazon is the following method the most efficient:
- Get List of All Parent Categories
- Find List of Sub Categories of the Parents (http://www.amazon.com/gp/site-directory/ref=nav_shopall_btn)
- From that list of all sub categories that I am interested in use the Product Advertising API to find the "browsenodes" of each.
- Use a recurring loop on each browse node until there are no more children nodes
- Use that list of categories with api's ItemSearch to return all of the ASIN's in that node
This would be run as a python script - due to the immense number of categories/subcategories/sub-sub-categories and ultimately ASIN's I know this will be a huge collection of product ASINs.
Is there a resource that already has this information? Or is the usage of Amazon's API the most efficient method?
