6

Looking for a database/databases of popular news headlines. These could be data from specific publications or public record, and could be either international or American. Ideally this would cover at least the last 60 years.

5

The New York Times has a web-based archive of news stories going back to 1851. Headlines are available without being a paid member. I'm sure other individual media companies will have a similar archive, but I'm only familiar with the NYTimes one.

After searching a time range, we notice that the URL of the results page can be used to create a scraping alogirthm.

http://query.nytimes.com/search/sitesearch/#/*/from19140101to19140131/

contains results from Jan. 1, 1914 to Jan. 31, 1914. You can even a summary of each article and download the full PDF!

Unfortunately, it's not a neat and clean database... Here is a sample scraping algorithm I wrote in order to get the Google results count from the HTML page (link). The important part is here (download and parse)

text = requests.get(url).text # get google html
m2 = re.search('About ([0-9,]+) results', text) # search for results

You may find that the number of searches is limited (so be gentle with the scraping).

2

It is not clear what you mean by popular news headlines.

Do you mean popular in the sense the article with that headline was shared a lot on social media? If yes, you can use the Event Registry. It provides access to over 100 million news articles and you can sort the articles based on how frequently they were shared on social media. In this way you can easily find the popular articles. You can use the Python library to freely access the data: https://github.com/gregorleban/EventRegistry/

The system also identifies events mentioned in the articles. In this way it is also easy to detect big events based on the number of different articles that write about them.

Event Registry frontend

0

These are compilations of news headlines by date over long periods of time sourced from 3 very different types of agencies:

ABC Australia 14 years (objective, accurate, reliable, Australia) : https://www.kaggle.com/therohk/million-headlines

The Examiner 6 years (crowdsourced, unreliable, clickbait, subjective) : https://www.kaggle.com/therohk/examine-the-examiner

Times Of India 17 years (factual, reliable, India) : https://www.kaggle.com/therohk/india-headlines-news-dataset

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.