Looking for a database/databases of popular news headlines. These could be data from specific publications or public record, and could be either international or American. Ideally this would cover at least the last 60 years.
-
Are you restricting yourself to any particular language?– ashes999Mar 9, 2014 at 4:12
-
No restrictions. Any leads would be useful.– yatakakaMar 9, 2014 at 21:11
-
1I have a short guide on how to access archived newspaper websites: opendata.stackexchange.com/a/1679/1511 (see The Internet Archive - Wayback Machine) but that only includes websites and not print media.– philshemMar 20, 2014 at 10:43
-
Here is the answer: opendata.stackexchange.com/questions/3744/…– Weijing Jay LinJan 26, 2016 at 21:23
Add a comment
|
3 Answers
The New York Times has a web-based archive of news stories going back to 1851. Headlines are available without being a paid member. I'm sure other individual media companies will have a similar archive, but I'm only familiar with the NYTimes one.
After searching a time range, we notice that the URL of the results page can be used to create a scraping alogirthm.
http://query.nytimes.com/search/sitesearch/#/*/from19140101to19140131/
contains results from Jan. 1, 1914 to Jan. 31, 1914. You can even a summary of each article and download the full PDF!
Unfortunately, it's not a neat and clean database... Here is a sample scraping algorithm I wrote in order to get the Google results count from the HTML page (link). The important part is here (download and parse)
text = requests.get(url).text # get google html
m2 = re.search('About ([0-9,]+) results', text) # search for results
You may find that the number of searches is limited (so be gentle with the scraping).
It is not clear what you mean by popular news headlines.
Do you mean popular in the sense the article with that headline was shared a lot on social media? If yes, you can use the Event Registry. It provides access to over 100 million news articles and you can sort the articles based on how frequently they were shared on social media. In this way you can easily find the popular articles. You can use the Python library to freely access the data: https://github.com/gregorleban/EventRegistry/
The system also identifies events mentioned in the articles. In this way it is also easy to detect big events based on the number of different articles that write about them.
These are compilations of news headlines by date over long periods of time sourced from 3 very different types of agencies:
ABC Australia 17 years (objective, accurate, reliable, aus, government, active) : https://www.kaggle.com/therohk/million-headlines
Times Of India 19 years (factual, reliable, ind, private, active) : https://www.kaggle.com/therohk/india-headlines-news-dataset
The Examiner 6 years (crowdsourced, unreliable, clickbait, subjective, usa, private, defunct) : https://www.kaggle.com/therohk/examine-the-examiner
