13

I'm looking for the materials of major newspapers, such as The New York Times, Washington Post, and The Economist. A random sample of their articles or headlines would suffice, but each newspaper has complications.

Some newspapers allow crawling.

Naturally, paid services search across newspapers and may do some export. See LexisNexis and Archives.gov.

Is there an open academic dataset of the newspaper content?

  • 1
    Years ago, I worked on a portal and we had an agreement to get Washington Post's current content via a NewsXML feed. (this was ~15 years ago.). Unlike RSS or Atom, we got the full article, not just an abstract / summary / first paragraph. (I would've actually prefered RSS, as we were just showing the titles + first paragraph, then linking back to them). This didn't allow us retrieve archived news, though. And it's possible that they've changed to just abstracts, as I don't remember it inserting 'read full article' on each item – Joe Apr 13 '15 at 12:23
  • 1
    Oh ... and if RSS is okay (and you'd just collect them going forward), I'd have sent you to the news tap of Syndic8, but it looks like something's screwed up in their web hosting. The front page is also at elsindi8.com , but all of the deeper links are broken. – Joe Apr 13 '15 at 12:31
  • I agree that this is really important. I have been looking for something like this for a while as well. As far as I can tell, there are sites that will provide real-time news through an api, and then you have to collect it yourself--such as twitter. Have not found good historical archives for newspapers though. – krishnab Apr 30 '15 at 2:41
7

Wikipedia: List of online newspaper archives

Many lack "official" interfaces and may restrict the use of materials (see their copyrights), but those that fit the requirements can be scraped with web services like import.io or Python's Scrapy:

(It also includes Google's scans of old newspapers.)

Library of Congress: Newspaper Archives, Indexes & Morgues

The links to a few dozens of newspaper archives with full-text articles. International newspaper archives included. Mostly historical data.

All Digitized Newspapers 1836-1922 by state, ethnicity, language:

Their API:

5

Material like this are rarely truly open because of copyright concerns. That said, there are academic news article corpora which can be obtained.

Reuters produced a corpus in 2000 which was later transferred to NIST. There are actually multiple corpora with hundreds of thousands of articles. More information about how to obtain them is on this page As far as I see, there is no cost, but you must apply for access.

Many academics access this kind of data through the Linguistic Data Consortium. This isn't open data, as a non-profit annual membership costs a few thousand dollars. But it is a major source for text corpora, including those built from news stories.

4

You can use Webhose.io API to integrate the data you are looking for. You can filter for news articles, blog posts, online discussions and much more. By default you get 1,000 monthly API requests for free, each request may return up to 100 posts, so you can access up to 100,000 articles a month for free.

2

It's not exactly "open academic dataset" but ProgrammableWeb keeps an API directory and has categories that may fit your request:

2

These are compilations of news headlines by date over long periods of time sourced from 3 very different types of agencies:

ABC Australia 14 years (objective, accurate, reliable, Australia) : https://www.kaggle.com/therohk/million-headlines

The Examiner 6 years (crowdsourced, unreliable, clickbait, subjective) : https://www.kaggle.com/therohk/examine-the-examiner

Times Of India 17 years (factual, reliable, India) : https://www.kaggle.com/therohk/india-headlines-news-dataset

2

The Event Registry can provide you access to news articles from last 3 years. The system currently holds over 450 million news articles. You can easily get the access to the data using the Python library available here: https://github.com/EventRegistry/. For each article you can get not only the original metadata but also the list of annotated concepts, categories, etc.

In addition to the news articles it can also provide you events - each thing that happens in the world and is reported in multiple news articles is considered to be an event. For each event you can find where it happened, when, who was involved, etc.

The provided API allows one to search for articles and events using various search criteria (keywords, concepts, categories, news source, event location, date, ...). The user can do per day 1000 free search requests.

enter image description here

  • github repo seems to be gone. :( – slashdottir May 14 at 19:26
  • sorry, the repo link has updated. I've updated the link also in this answer – Gregor Leban May 15 at 20:08
  • does not seem to be exactly open data... – famargar May 27 at 7:19
2

There is also the All The News dataset from Kaggle. It has about 140K articles from major USA newspapers and news sites. Quoting from the Kaggle site:

The publications include the New York Times, Breitbart, CNN, Business Insider, the Atlantic, Fox News, Talking Points Memo, Buzzfeed News, National Review, New York Post, the Guardian, NPR, Reuters, Vox, and the Washington Post. Sampling wasn't quite scientific; I chose publications based on my familiarity of the domain and tried to get a range of political alignments, as well as a mix of print and digital publications. By count, the publications break down accordingly.

The data primarily falls between the years of 2016 and July 2017, although there is a not-insignificant number of articles from 2015, and a possibly insignificant number from before then.

I am using it for a project I am currently working and it is quite good (quite clean text).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.