Historical weather data

Question

I want to get historical weather data (Winter 2014) of temperature, humidity, air-pressure, wind_speed, wind direction, rain in specific latitude/longitude.

Is there any API that I can use to get these informations.

Answer 1

For international and historical data, and for a modest number of requests per day, I personally recommend the Wunderground API. Once you register, you can get 500 free requests per day.

The URL for historical data will look like this:

http://api.wunderground.com/api/Your_Key/history_YYYYMMDD/q/CA/San_Francisco.json

I've posted a sample code (python 2.7) that you can use (and improve!) - LINK. I would run this code every day, just changing the year (currently it's set for 2013). The output of the code is a CSV file, but you can store the JSON and/or parse as needed.

Answer 2

I have written some sample code for directly building a CSV starting with a given date and ending with a given date: https://github.com/joshmalina/pollution/blob/master/notebooks/Build_historical_weather_data.ipynb

The city is currently set to Beijing, but you can change that easily. The data will also be cleaned of null values.

Answer 3

For Canada, you can download historical data by city in bulk csv or xml files from Environment & Climate Change Canada.

The example provided here uses wget to download all available hourly data for Yellowknife A, from 1998 to 2008, in .csv format

for year in `seq 1998 2008`;
    do for month in `seq 1 12`;
    do wget --content-disposition "http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=1706&Year=${year}&Month=${month}&Day=14&timeframe=1&submit= Download+Data" ;
    done;
done

WHERE;
• year = change values in command line (seq 1998 2008)
• month = change values in command line (seq 1 12)
• format= [csv|xml]: the format output
• timeframe = 1: for hourly data
• timeframe = 2: for daily data
• timeframe = 3 for monthly data
• Day: the value of the "day" variable is not used and can be an arbitrary value
• For another station, change the value of the variable stationID
• For the data in XML format, change the value of the variable format to xml in the URL.

You can grab a list of stations from this csv or search for a station

Answer 4

If you solve ML task and want to try weather historical data as features, I recommend you to try python library upgini for smart enrichment. It contains 12 years history weather data by 68 countries.

My code of usage is following:

%pip install -Uq upgini
from upgini import SearchKey, FeaturesEnricher
from upgini.metadata import CVType, RuntimeParameters

## define search keys
search_keys = {
    "Date": SearchKey.DATE, 
    "country": SearchKey.COUNTRY,
    "postal_code": SearchKey.POSTAL_CODE
}

## define X_train / y_train
X_train=df_prices.drop(columns=['Target'])
y_train = df_prices.Target

## define Features Enricher
features_enricher = FeaturesEnricher(
search_keys = search_keys,
cv = CVType.time_series
)

X_enriched=features_enricher.fit_transform(X_train, y_train, calculate_metrics=True)

As a result you will get dataframe with new features with non-zero feature importance on your target variable, such as temperature, wind speed etc

Web: https://upgini.com GitHub: https://github.com/upgini