The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

10

There's a big difference between “available to the public” and “belonging to the public”. By accessing a privately-owned website, you are accepting its terms and conditions. These conditions typically preclude scraping and aggregation. Here's an example from Amazon's (Canada) Licence and Access terms: Subject to your compliance with these Conditions of Use ...


6

It sounds like you want Consumer Price Indices for Food and Beverage for various metropolitan areas. The Bureau of Labor Statistics calculates just such indices: http://download.bls.gov/pub/time.series/cu/cu.txt Metropolitan Areas area_code area_name A101 New York-Northern New Jersey-Long Island, NY-NJ-CT-PA1 A102 Philadelphia-...


6

My best guess is World petrol prices on MyTravelCost.com. They give you a large, configurable bar chart of gasoline prices. However, their source description can be described vague at best: The data are drawn from a variety of sources including official government materials, oil companies, online resources specializing in gas prices, and others. These ...


5

The "FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS - Statistics Division" (link) provides historical data for commodity prices and agricultural production. Production Prices You can download for a single country or for all countries. Some data sets go back to at least the 1960s. You'll have to investigate which data sets may be best suited to ...


5

http://www.isbndb.com has a good API that uses XML calls/responses to get new/used prices from Amazon.com as well as a number of other sites like abebooks, half.com, etc...


5

www.keepa.com is a similar website to camelcamelcamel. I cannot find a way to download the data but maybe you will. In addition, Terapeak provide a free package with limited api calls (500/months) where you can find historical prices.


4

The data you're looking for is available here -: https://www.datazar.com/project/pe1c88ca8-57a8-48b8-89a6-89083da7fc3c It has historical data from 1918, it is free and open for use. P.S- I work for datazar, an open source data library, where this file is hosted.


4

You are looking for the variables in MEPS (the Medical Expenditure Panel Survey from AHRQ) with the text slf in them. There are both utilization-level files as well as a consolidated person-level file available every year. http://www.asdfree.com/search/label/medical%20expenditure%20panel%20survey%20%28meps%29


4

Index Mundi has a lot of commodity data (present and historical) available to the public. But I am not sure if you would find toilet paper on the list (lol) - it does not exactly fit the definition for a commodity: "a raw material or primary agricultural product that can be bought and sold, such as copper or coffee." http://www.indexmundi.com/commodities/


4

You can get release dates of versions/iterations at http://en.wikipedia.org/wiki/Adobe_Creative_Suite but I think you'll need to use a combination of Google and The Wayback Machine by archive.org to find pricing. Also see http://www.computerworld.com/article/2517120/enterprise-applications/adobe-creative-suite--the-history.html


4

You can also scrape (or better yet contact and ask the folks at) https://thetracktor.com which seems to have some great historical data. EDIT - Adding example: https://thetracktor.com/ajax/prices/?id=881549&days=90 is the JSON response powering the graph at https://thetracktor.com/detail/B0054JJ0QW/, for example.


3

This might be helpful for other people if not for OP. The problem here is that US Census data is based on census tracts, not zip codes. So for the US here is HUD "crosswalk" data from www.Huduser.gov that converts zip codes to census tracts, and vice versa. This will get you part way there, and may be very useful for other people. It's also recent, as of ...


3

Living Wage Calculator by MIT provides cost of living data down to locality, which you could mesh into zip codes, but since zip codes are not unique to a locality, you'd have to give or take something. perhaps use census zcta's? I'm not 100% on that, that's just the first thing that comes to mind. I don't see any bulk download options, but they do use a ...


3

Medicare has a public dataset that may provide the information you are looking for: Medicare Provider Utilization and Payment Data: Physician and Other Supplier As part of the Obama Administration’s efforts to make our healthcare system more transparent, affordable, and accountable, the Centers for Medicare & Medicaid Services (CMS) has ...


3

Try looking at Numbeo. It is a crowd source project on user contributed real-time data. The cost-of-living section has data on gasoline prices. http://www.numbeo.com/cost-of-living/ Creative Commons Attribution-Sharealike 3.0 Unported License (CC-BY-SA) and GNU Free Documentation License (GFDL)


3

The World Bank did this kind of project in 2012-2013: http://data.worldbank.org/data-catalog/crowd-sourced-price-collection The objective of the pilot was to study the feasibility of crowd-sourced price data collection. Non-professional price collectors used personal computers and mobile phones for collecting data and entering it in a multilingual ...


3

Gas Buddy, the Price of Weed website (SE does not allow a link) and Craigslist all come to mind.


3

Another source of this data, just discovered - from the World Bank: - petrol/gas - diesel


3

If you're looking for how prices vary over time, in the U.S., there's the Consumer Price Index, which includes various categories of food prices. I've never dug into their raw data, so I don't know how much detail they make available (eg, comparisons by state or other region).


3

The CPI indices from the Bureau of Labor and Statistics is probably what you're looking for. While it doesn't tell you what the price of 12 ounces of Coca Cola is, it does tell you what the average price of, say, "carbonated beverages" is over time. It will take some munging, but the raw data is located here


3

Check out these databases on Quandl: Australian Bureau of Statistics: https://www.quandl.com/data/AUSBS?keyword=construction Organisation for Economic Co-Operation and Development with data on Australia: https://www.quandl.com/data/OECD?keyword=construction%20cost%20australia National Institute of Statistics and Economic Studies with data on France: https:...


3

Zillow is an online real estate database company and offers research data which is aggregated, and you may find meets some of your needs. http://www.zillow.com/research/data All data accessed and downloaded from this page is free for public use by consumers, media, analysts, academics etc., consistent with our published Terms of Use. Proper and clear ...


2

During the best city contest last year the Economist published a Worldwide Cost of Living Index which include food prices for multiple cities across the globe. Data are available under Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) license. Currently the Economist website is down, but I am sure you can find updated version on their website (I will try to ...


2

I've found a couple of sources of retail prices in open access: http://www.pricestats.com/ http://data.worldbank.org/data-catalog/crowd-sourced-price-collection (same here https://app.enigma.io/table/org.worldbank.crowd-sourced.price.collection.csv) But the World Bank covers only 8 developing countries. And PriceStats shares only US prices with a 10-day ...


2

The World Bank provides data, but it is aggregated on the yearly level. Their source is German Agency for International Cooperation (GIZ). Pump price for gasoline (US$ per liter) Fuel prices refer to the pump prices of the most widely sold grade of gasoline. Prices have been converted from the local currency to U.S. dollars. There is a download button ...


2

Wikipedia has the information for all countries: https://en.wikipedia.org/wiki/Gasoline_and_diesel_usage_and_pricing#Typical_gasoline_prices_around_the_world Creative Commons Attribution-ShareAlike License There is also http://www.globalpetrolprices.com/gasoline_prices/ but it has a much more restrictive BY-NC-ND license.


2

The US Bureau of Transportation Statistics provides various stats and datasets for historical airline ticket prices. General overview, datasets and statistics Airfare data From the Route Fares page: Route fares If you go through the PDF report (2012 to Q1 2014), you can find the corresponding raw data (CSV/Excel) for each table.


2

A source at the US federal government level is GSA Advantage! which allows schedule holders (government contractors) to list products and services that government agencies can purchase much like an online shopping cart. This is by no means clean or normalized data and runs into the issues described in the comment thread above about discounts/sales. Another ...


2

How about a government jurisdiction's ledger? This would be more granular than for each contract and instead be for each check written. Some examples: State of Iowa: Ledger data at https://data.iowa.gov/d/9sag-ywd8; Exploratory UI at http://checkbook.iowa.gov/ City of Danvilla, VA: Ledger data at https://data.danvilleva.gov/d/2b9c-d6ip; Exploratory UI at ...


2

I think it depends of the nature of the site, not only the terms and conditions. In the case of privately owned health medicine information, it could be an exception because of the nature itself of information or in the case of government site with all rights reserved default license, it could be argued that there is enough legal ground to try particular ...


Only top voted, non community-wiki answers of a minimum length are eligible