Barry, could you abuse this site's "answer your own question" feature to create a community wiki answer for sources of weather data (both current and historical), since it gets asked so often?
4I see answers creeping in with data from models. Can we please distinguish real data and model data? (I suggest limiting this question to observational data only).– user4293Nov 28, 2016 at 10:04
3We also need (with each link) good descriptions of the data in the set and its historic period. Can I suggest additions in the format [precipitation, wind speeds, temperatures][1960-current]? Maybe add time resolution as well?– user4293Nov 28, 2016 at 10:06
2Note that there is significant overlap with questions on Earth Science. I have not investigated if they have a canonical/community question over there, it might be wise to team up. Start a proposal in meta on both sites?– user4293Nov 28, 2016 at 10:08
.. and geographical area (can no longer edit my second comment).– user4293Nov 28, 2016 at 10:12
@JanDoggen the distinction between 'real data' and 'model data' is not always clear (b/c, e.g. processing of radar requires models, as do historical reprocessing and reanalysis); 'station data' would be closest to 'real' although this is generally re-processed to remove artifacts and fill gaps ... using models) perhaps a better distinction would be 'best guess at historical reality' vs. model predictions (or historical vs future)?– David LeBauerNov 28, 2017 at 17:17
This list is (and may always be) incomplete. Please add to it when you can.
Per minute data: ftp://ftp.ncei.noaa.gov/pub/data/asos-onemin
NOAA's ISD data (goes back to ~1900 in some cases):
https://www.ncdc.noaa.gov/isd (home page)
ftp://ftp.ncdc.noaa.gov/pub/data/noaa/ (direct link to data, some stations include multiple report per hour in some cases)
https://www1.ncdc.noaa.gov/pub/data/noaa/isd-lite/ (direct link to "lighter" version of data-- hourly observations only)
Global Historical Climatology Network (GHCN) observations, some dating back to the mid 1700s:
http://mesowest.utah.edu/ includes solar radiation and snowfall accumulation, which some other sites don't. Sample:
The "raw" data as it comes into NOAA from airports, surface weather stations, boats, and buoys:
You can read more about these "cycle" files at: http://www.nws.noaa.gov/tg/datahelp.php Note these files are updated every few minutes and have not been curated, so data can be inaccurate. Reports in these cycle files are sometimes updated (and thus invalidated) in later cycle files.
To translate station codes to physical locations:
- The file "isd-history.csv" in ftp://ftp.ncdc.noaa.gov/pub/data/noaa/
- Commercial source (but some portions are free): http://www.weathergraphics.com/identifiers/
Well known sites:
Thank you for the immensely helpful link. I am planning on using this link for historical hourly temperatures: ftp.ncdc.noaa.gov/pub/data/noaa It contains yearly folders, e.g. 2017 within which each station has its own file, e.g. 010010-99999-2017 The only question I have: how to get the location of the station, 010010 in this case? Lat and long is fine, even zip code would be OK. I tried to find such a list but was not successful. Dec 19, 2018 at 20:23
1@KevinP. I've added a paragraph near the end re sources to identify station "metadata". For your use case, isd-history.txt should work– user3856Dec 19, 2018 at 23:03
Thank you again, Barry. I was able to easily use wget to get this path: ftp.ncdc.noaa.gov/pub/data/noaa/isd-history.csv Dec 23, 2018 at 1:12
A further question: Does anyone know if the hour field is for the local time zone of the weather station (I would assume so) and if it has daylight savings already applied to it? I read through all the documentation but it doesn't address it one way or the other. Dec 29, 2018 at 1:26
1@KevinP.I'm pretty sure it 's all UTC. If you google
isd "utc" site:.gov(as quoted) and then look at the cached version of the pages (the main ones are down due to US govt shutdown), you might find a canonical confirmation (I found a lot that suggests all times are UTC, but nothing canonical for isd-lite specifically)– user3856Dec 29, 2018 at 19:53
Get the data from BigQuery:
#standardSQL SELECT name, stn, a.wban, TIMESTAMP(CONCAT(year, '-', mo, '-', da)) date, temp, prcp FROM `bigquery-public-data.noaa_gsod.gsod2*` a JOIN ( SELECT * FROM `bigquery-public-data.noaa_gsod.stations` WHERE country='US' AND name LIKE '%HONOLULU%' AND state LIKE 'HI' LIMIT 1000 ) b ON a.stn=b.usaf AND a.wban=b.wban WHERE prcp!=99.99 ORDER BY name, date
Related: Finding the rainiest US cities
A remodel of weather since 1979, based on many many measurements. The resulting dataset contains daily to hourly values around the globe for many meteorological and climatological parameters such as temperatures, water content, wind, radiation and various fluxes.
The most up to date version of this dataset is ERA 5: https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5
A project similar to ERA 5.
http://www.esrl.noaa.gov/ The Earth System Research Laboratory, with data and information or aerosols, as well as other temperature, precipitation etc. Within this organisation:
(closed data) A commercial service that hosts a lot of high detail weather data, and allows a limited number (1000/day) of requests for free: https://darksky.net/. Darksky has been acquired by Apple and their API is no longer available.
There's a good list of weather data sources on this excellent website created by Robin Wilson, which I duplicate here in case that site ever goes away:
- WorldClim: Climate data for past, present and predicted future conditions. Includes temperature (min, max, mean) and precipitation. 1km x 1km resolution.
- NCAR GIS Climate Change Scenarios: Lots of data from the National Centre for Atmospheric Research models, including data used by the IPCC in their reports. Registration required
- CRU Climate Datasets: Various datasets including temperature, precipitation, pressure, drought, paleoclimate and others, from the (infamous) Climate Research Unit at the University of East Anglia. High and low resolution datasets available.
- Downscaled GCM Portal: A wide range of downscaled (that is, higher resolution data) created from the outputs of a wide range of GCMs. Covers the majority of important climate variables.
- European Climate Assessment and Dataset: Gridded observation data across Europe, including cloudiness, temperature, precipitation, humidity, pressure, snow and sunshine.
- Global Potential Evapotranspiration and Aridity Index: Potential Evapotranspiration data and indices of aridity at 30 arc-second resolution in raster format.
- GlobAerosol: Aerosol data in raster format (provided as NetCDF files), including estimations of AOD at 550nm and the Angstrom coefficient.
- Global Aerosol Climatology Project: Monthly averages of Aerosol Optical Thickness and the Angstrom exponent from 1981 to 2006.
- Satellite Application Facility on Climate Monitoring: Provides near real-time and retroactively-generated datasets of cloud cover, type and temperature, surface radiation budget and temperatures, amongst others.
- Climate Analysis Indicators Tool: Carbon Dioxide emissions data by country, and by US state, including measures of adaption.
- International Satellite Cloud Climatology Project: Monthly averages of a number of cloud variables including cloud cover (and associated breakdown by frequency), cloud top temperature, cloud water path, cloud optical thickness etc.
- Climatological Database for the world's oceans (CLIWOC): Data compiled from ships logs compiled during voyages between 1750 and 1850. Includes various meteorological observations, broken down by date, ship and year.
- IRI/LDEO Climate Data Library: A wide range of over 300 datasets from various climate models and datasets.
- World Ozone and Ultraviolet Radiation Data Center: Extensive set of data collected from many stations across the world, recording ozone information (from ozonesonde ascents) and ground-level ultraviolet irradiance.
- Gridded climatic data for North America, South America and Europe: A huge range of climatic data at 1km and 4km resolution, derived from various models, including temperature, precipitation, snow and derived variables such as water deficit and continentality.
There are hundreds more links to geospatial data on other topics on the main site, as well as a link to donate and keep that site alive.
Also available through the Wayback Machine (added on April 5, 2019).
The resources below are not from Robin Wilson's site:
The Köppen-Geiger climate classification, which divides the Earth's land area into 30 climatological regions based on location, temperature, and precipitation. A high-resolution map is available here
There are several modified versions of the Köppen-Geiger climate classification. One of the more popular modifications is the Trewartha climate classification
Holdridge Life Zones, an alternative climate classification based on precipitation, humidity, and potential evapotranspiration ratio.
Since this answer seems very climate-focused, could you add the Koeppen-Geiger classification scheme: koeppen-geiger.vu-wien.ac.at/present.htm– user3856Jul 11, 2019 at 17:25
@BarryCarter Thanks for the suggestion, I added it to the list. Can you add a description?– cskJul 11, 2019 at 17:30
Done (+ added a couple more sources + made it clear these sources are not from Wilson's site). Edit will be invisible until peer-reviewed, though.– user3856Jul 12, 2019 at 4:18
I recently found this website for downloading weather data: https://rp5.ru/ which is the website of Reliable Prognosis, a company from St-Petersburg. The website is mainly dedicated to weather forecast, but you can find historical weather data (weather archive). There are interesting features for historical weather data:
- Available data from "8400 SYNOP weather stations and 5200 METAR weather stations" (according to https://rp5.ru/docs/about/en).
- Hourly, 3-hourly or daily data available (depending on the station).
- Many weather variables available : temperature, precipitation, pression, cloud cover, wind velocity & direction, ... (depending on the stations) + observations (textual information).
- A nice stuff - data is very frequently updated: data from METAR stations appears on the website 10 min. after recordings.
- Data licence? I did not find information about the data licence.
I've used data from this website for one station in France + had a quick look at other ones in Belgium. If someone else has more experience with this website, please comment!
I have recently worked on GWeather a command-line tool/library that helps users to work with climate data from all over the world. The main goal is to be able to work on a dataset where data from PSL will be presented in a friendly way combining location names from simplemaps.
The application supports the next features:
- Download climate datasets from PSL
- Download cities/countries dataset from simplemaps.com
- Merge/join above datasets by location's coordinates
- Export merged data in several formats
Comments remarks and pull requests are all welcome.