I'd like to know an estimated amount of open data over the web. This is required for a research proposal that I'm working on.
Some projects such as the Open Data Monitor aim to answer this question. For convenience, I split the answer into four sections.
The amount of open data released by governments is reasonably documented. Perhaps not in the sense of the exact amount of Gb.
- The Open Data Index by the Open Knowledge Foundation
- The Open Data Barometer by the The Web Foundation & The Open Data Institute.
- Catalogues of datasets such as datacatalogs.org or the World Bank.
This is much harder to quantify. Indicators are:
- The GovLab are running a survey, the Open Data 500, analysing companies.
- Data about businesses is published with a dual-licence by OpenCorporates. At the moment the database contains over 60 million entities.
- Many businesses publish data, often via an API. However, in most cases not as open data. An example is Twitter, where the licence and availability of the "hose" is substantially limited.
Institutions are publishers of some of the largests datasets on the web. For example:
- The 1,000 Genome project is one of the largest open datasets with an extreme size of 260Tb.
- The Tiny Images dataset consists of almost 80 million images stored in the form of large binary files. They are used in an academic paper on scene recognition from 2008.
- Researchers at the Measurement Lab (M-Lab), who publish over 750Tb of data under a CCZero licence.
The amount of personal data that individuals make available on the web is, to the best of my knowledge, negligible.
I would suggest characterizing the availability not by the quantity of data but by the expansion of open data portals. Here are some suggestions:
Visit the various open data catalogs which list (government and non-government) open data portals.
A. Estimate the number of open data portals that are available.
B. Look at when they were launched (DataCatalogs.org lists launch dates) and calculate a rate of expansion. (Jeanne Holm is one of the curators)
C. Sample portals at Federal, State and Municipality level to get an idea of the size and types of data that is made available.
D. Compare availability/expansion in various geo-economic regions (Northern America, Latin America, Europe, Asia, Northern Africa/Middle East, ...)
E. Research press releases of government iniatives on open data, open data initiaative, transparency data. E.g., Obama's Open Data Initiative 2009, and followup initiatives in 2012 and 2013
F. Research the number and type of 3rd party and open source tools and support infrastructure for open data.
G. Research the various emerging standards for accessing, organizing, format and layout. Compare access via direct dataset download (e.g., CSV files) vs. APIs.
There is a good list of catalogs available at Collect a list of open data systems
You can find related resources at OpenGeoCode, which I am a co-founder.