Average Monthly temperatures for cities around the world

Question

I am looking to find the average historical temperature in april (not the average temperature in april of 1988, for example) in different cities (about 400 cities across the world) and store that data in a format that is manageable in python. I've found the data in weatherbase.com but I haven't found a way to collect the data automatically (I've found one api for weatherbase written in python but it doesn't have much documentation) Does someone know a not-that-complicated way to collect the data from weatherbase or other website?

Answer 1

If there is no well documented API for weatherbase.com, it might be interesting to scrape the information. Here is the list of cities the site contains with their URL. You can select some cities and paste their rows into a csv file. This Python 2 script will read the .csv and extract the average temperature in February for each city. The code is probably very poorly written, but it is functional. Feel free to improve it.

#!/usr/bin/python

import csv
import requests
from bs4 import *

url_list=[]
with open('my_cities_list.csv', 'rb') as f:
    reader = csv.reader(f, delimiter='\t')
    for row in reader:
        print(row[0])
        url_list.append(row[0])

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.\
8",
    "accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
    "accept-encoding": "gzip,deflate,sdch",
    "accept-language": "en-US,en;q=0.8",
}

data=[]
for url in url_list:
    try:
        response = requests.get(url, headers)
        soup = BeautifulSoup(response.content, 'lxml')
        february_average = soup.select('.data')[2].getText().encode("utf-8")
        city=soup.select('h1')[0].getText().encode("utf-8")
        result=city + "," + february_average
        print(result)
        data.append(result)     
    except Exception as e:
       print(e, "with ", url)
       pass

with open('results.txt', 'w') as f:
    f.write( '\n'.join( data ).encode("utf-8") )

EDIT : Python 3 version with better results in csv

import re
import requests
import csv
from bs4 import BeautifulSoup

url_list = []
with open('test.csv', 'rt', encoding="utf8") as f:
    reader = csv.reader(f, delimiter='\t')
    for row in reader:
        print(row[0])
        url_list.append(row[0])

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.\
8",
    "accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
    "accept-encoding": "gzip,deflate,sdch",
    "accept-language": "en-US,en;q=0.8",
}

data = [["city","country","february_average"]]

for url in url_list:
    try:
        response = requests.get(url, headers)
        soup = BeautifulSoup(response.content, 'lxml')
        february_average = soup.select('.data')[2].getText()
        city = soup.select('h1')[0].getText()
        result = str(city) + "," + str(february_average)
        data.append(result.split(","))
    except Exception as e:
        print(e, "with ", url)
        pass

with open ('results.csv','w') as file:
   writer=csv.writer(file, delimiter=',')
   for row in data:
      print(row)
      writer.writerow(row)