3

I am looking to find the average historical temperature in april (not the average temperature in april of 1988, for example) in different cities (about 400 cities across the world) and store that data in a format that is manageable in python. I've found the data in weatherbase.com but I haven't found a way to collect the data automatically (I've found one api for weatherbase written in python but it doesn't have much documentation) Does someone know a not-that-complicated way to collect the data from weatherbase or other website?

  • 1
    Have you tried this API ? – Ettore Rizza Feb 13 '17 at 20:20
  • That api doesn't give me the historical averages... – leite0407 Feb 14 '17 at 0:29
4

If there is no well documented API for weatherbase.com, it might be interesting to scrape the information. Here is the list of cities the site contains with their URL. You can select some cities and paste their rows into a csv file. This Python 2 script will read the .csv and extract the average temperature in February for each city. The code is probably very poorly written, but it is functional. Feel free to improve it.

#!/usr/bin/python

import csv
import requests
from bs4 import *

url_list=[]
with open('my_cities_list.csv', 'rb') as f:
    reader = csv.reader(f, delimiter='\t')
    for row in reader:
        print(row[0])
        url_list.append(row[0])

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.\
8",
    "accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
    "accept-encoding": "gzip,deflate,sdch",
    "accept-language": "en-US,en;q=0.8",
}

data=[]
for url in url_list:
    try:
        response = requests.get(url, headers)
        soup = BeautifulSoup(response.content, 'lxml')
        february_average = soup.select('.data')[2].getText().encode("utf-8")
        city=soup.select('h1')[0].getText().encode("utf-8")
        result=city + "," + february_average
        print(result)
        data.append(result)     
    except Exception as e:
       print(e, "with ", url)
       pass

with open('results.txt', 'w') as f:
    f.write( '\n'.join( data ).encode("utf-8") )

EDIT : Python 3 version with better results in csv

import re
import requests
import csv
from bs4 import BeautifulSoup

url_list = []
with open('test.csv', 'rt', encoding="utf8") as f:
    reader = csv.reader(f, delimiter='\t')
    for row in reader:
        print(row[0])
        url_list.append(row[0])

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.\
8",
    "accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
    "accept-encoding": "gzip,deflate,sdch",
    "accept-language": "en-US,en;q=0.8",
}

data = [["city","country","february_average"]]

for url in url_list:
    try:
        response = requests.get(url, headers)
        soup = BeautifulSoup(response.content, 'lxml')
        february_average = soup.select('.data')[2].getText()
        city = soup.select('h1')[0].getText()
        result = str(city) + "," + str(february_average)
        data.append(result.split(","))
    except Exception as e:
        print(e, "with ", url)
        pass

with open ('results.csv','w') as file:
   writer=csv.writer(file, delimiter=',')
   for row in data:
      print(row)
      writer.writerow(row)
  • This has been a very useful answer for me, but there is an issue with your list of cities: it is missing at least one country entirely, Mexico. It might be missing more. How did you create it? – equant Dec 16 '18 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.