Dataset of major newspapers content

Question

I'm looking for the materials of major newspapers, such as The New York Times, Washington Post, and The Economist. A random sample of their articles or headlines would suffice, but each newspaper has complications.

• The New York Times has an article search API that gives access to the newspaper's archives. There's a Python wrapper for it. The NYT allow frequent queries ( > 5,000 calls per day) after contacting them (non-commercial use only).

• The Guardian grants free API access to 1.7M articles.

Some newspapers allow crawling.

Naturally, paid services search across newspapers and may do some export. See LexisNexis and Archives.gov.

Is there an open academic dataset of the newspaper content?

Answer 1

Wikipedia: List of online newspaper archives

Many lack "official" interfaces and may restrict the use of materials (see their copyrights), but those that fit the requirements can be scraped with web services like import.io or Python's Scrapy:

• http://en.wikipedia.org/wiki/Wikipedia:List_of_online_newspaper_archives

(It also includes Google's scans of old newspapers.)

Library of Congress: Newspaper Archives, Indexes & Morgues

The links to a few dozens of newspaper archives with full-text articles. International newspaper archives included. Mostly historical data.

• http://www.loc.gov/rr/news/oltitles.html

All Digitized Newspapers 1836-1922 by state, ethnicity, language:

• http://chroniclingamerica.loc.gov/#tab=tab_newspapers

Their API:

• http://chroniclingamerica.loc.gov/about/api/

Answer 2

Material like this are rarely truly open because of copyright concerns. That said, there are academic news article corpora which can be obtained.

Reuters produced a corpus in 2000 which was later transferred to NIST. There are actually multiple corpora with hundreds of thousands of articles. More information about how to obtain them is on this page As far as I see, there is no cost, but you must apply for access.

Many academics access this kind of data through the Linguistic Data Consortium. This isn't open data, as a non-profit annual membership costs a few thousand dollars. But it is a major source for text corpora, including those built from news stories.

Answer 3

You can use Webhose.io API to integrate the data you are looking for. You can filter for news articles, blog posts, online discussions and much more. By default you get 1,000 monthly API requests for free, each request may return up to 100 posts, so you can access up to 100,000 articles a month for free.

Answer 4

There is also the All The News dataset from Kaggle. It has about 140K articles from major USA newspapers and news sites. Quoting from the Kaggle site:

The publications include the New York Times, Breitbart, CNN, Business Insider, the Atlantic, Fox News, Talking Points Memo, Buzzfeed News, National Review, New York Post, the Guardian, NPR, Reuters, Vox, and the Washington Post. Sampling wasn't quite scientific; I chose publications based on my familiarity of the domain and tried to get a range of political alignments, as well as a mix of print and digital publications. By count, the publications break down accordingly.

The data primarily falls between the years of 2016 and July 2017, although there is a not-insignificant number of articles from 2015, and a possibly insignificant number from before then.

I am using it for a project I am currently working and it is quite good (quite clean text).

Answer 5

It's not exactly "open academic dataset" but ProgrammableWeb keeps an API directory and has categories that may fit your request:

• News Services

• Media

• Magazines

• maybe more...

Answer 6

The Event Registry can provide you access to news articles from last 3 years. The system currently holds over 450 million news articles. You can easily get the access to the data using the Python library available here: https://github.com/EventRegistry/. For each article you can get not only the original metadata but also the list of annotated concepts, categories, etc.

In addition to the news articles it can also provide you events - each thing that happens in the world and is reported in multiple news articles is considered to be an event. For each event you can find where it happened, when, who was involved, etc.

The provided API allows one to search for articles and events using various search criteria (keywords, concepts, categories, news source, event location, date, ...). The user can do per day 1000 free search requests.

Answer 7

These are compilations of news headlines by date over long periods of time sourced from 3 very different types of agencies:

ABC Australia 17 years (objective, accurate, reliable, aus, government, active) : https://www.kaggle.com/therohk/million-headlines

Times Of India 19 years (factual, reliable, ind, private, active) : https://www.kaggle.com/therohk/india-headlines-news-dataset

The Examiner 6 years (crowdsourced, unreliable, clickbait, subjective, usa, private, defunct) : https://www.kaggle.com/therohk/examine-the-examiner