Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It only takes a minute to sign up.
Sign up to join this community
UN Comtrade database, the major source of bilateral trade data online, has a free query tool:
But it has limits on "query complexity", which makes bulk downloads burdensome.
Does anyone have this entire database as a single file to make panel analysis easier?
You can query the entire database through the legacy interface. BUT, it still restricts the download to max of 50,000 records. You will need to breakup your queries to get what you want
http://comtrade.un.org/db/dqBasicQuery.aspx
This shows the contact info if you want to get it directly from them. Note, I queried for the entire database:
The query will return 456722204 records.
It's not recommended to view it on screen (1000 records limit). Please modify your query (or select additional selection from below) or click download.
It's not possible to have direct download if the records are more than 50,000. You could split the query and make multiple downloads within the limits or you could submit the query to batch processing and retrieve it later or you could contact us.
* Estimated file size: 55,892.03 MB.
It is possible to access and download the data via an API. The queries you can submit are also restricted, but it is possible to facilitate and automate the extraction by using some programming.
You can write a script to download the data in chunks and assemble them afterwards. Here is an example of how you could do it in R. A first chunk covers the years 2009-2011 and a second chunk the years 2012-2014. Here I have used copy-paste, but if you have more voluminous and more complex requests, you can use loops as well.
download.file("http://comtrade.un.org/api/get?fmt=csv&ps=2009,2010,2011&r=all&p=0&cc=TOTAL", "data1.csv", method = "auto", quiet=FALSE)
download.file("http://comtrade.un.org/api/get?fmt=csv&ps=2012,2013,2014&r=all&p=0&cc=TOTAL", "data2.csv", method = "auto", quiet=FALSE)
data1 <- read.table("data1.csv", header = T, sep = ",")
data2 <- read.table("data2.csv", header = T, sep = ",")
allthedata <- rbind(data1, data2)
I've not tried it, but perhaps it'll help someone:
An alternative, one-file dataset on bilateral trade from The Correlates of War project:
May serve as a quick replacement for Comtrade.
