15

I am working on a data-visualization web application that looks at trends in American news media coverage over time. It takes a keyword (or keywords), a date range, and a time increment as parameters and generates a line graph and data set of how often that keyword appears in news articles during the given time period (with one data point for each time increment- either a month or year). So if you search "Mitt Romney" in 2012, you'll see a line that gradually fluctuates during primary season and peaks around the November election, before falling off, and a table with each data point's value and date range.

Right now I am manually scraping this information (no API) from an external web site. However this is inefficient, slow, and the site's ToS are fairly restrictive. I would ideally like to swap out this backend with a new data source, but I am having trouble finding sources that fit my needs.

I need a data source that:

  1. Shows the number of results of a search. This can be subtle or a non-central function. Search engines or APIs that show things like "Results 1-10 of 1242" fulfill this criteria
  2. Supports specific date ranges. So I can get results from "April 1 2013 to April 1 2014" rather than just "last 30 days" or something
  3. Is cheap or free, and has non-restricted ToS.

This has proven fairly complicated. I've looked at the following and found:

  • Google News API is deprecated.
  • Bing News API seems like it would fit this functionality reasonably well, but they have a low per-second query limit, their free monthly limit is low, and their prices to get higher limits are oppressively high.
  • Faroo's API is the closest to usable that I've found so far. They support news search by keyword, are free, and have a high query limit. The number of articles that are returned by a query is capped at 10, but it shows the total number of results for the search criteria, which is exactly what I'm looking for. However Faroo does not allow you to search by date; articles returned store their publication dates, but since only 10 articles are returned, this does not allow for effective use of frequency over time data, which is critical.
  • I've investigated a number of other smaller search engines, but they all lack critical features.

This is a fairly stringent set of requirements and I've therefor had trouble finding an actual API that fits the bill. Is anyone aware of APIs or databases that I can use that support the above functionality? Ideally I need a free way of pairing keyword, date range and article frequency, but haven't found any broadly aggregating data source that I can use.

Any suggestions greatly appreciated.

migrated from stats.stackexchange.com Oct 2 '14 at 2:34

This question came from our site for people interested in statistics, machine learning, data analysis, data mining, and data visualization.

  • 1
    You might contact the folks at emergent to ask what their data feed is. (it's a fairly recent project, trying to track if 'news' is just an echo-chamber of rumors or actually sustantiated/verified) – Joe Oct 2 '14 at 14:31
  • 1
    Thanks for the suggestion. I contacted the Emergent guys, and apparently while they have some algorithmic tracking, they curate tracked rumors manually. – UpQuark May 28 '15 at 14:45
  • 1
    chroniclingamerica.loc.gov/ocr is a bit excessive, but useful for OCR'd newspapers from "ancient" times – Barry Carter Oct 1 at 16:36
  • @BarryCarter that sounds very cool, I'd post that as an answer so it's more visible. There don't seem to be a lot of aggregate sources like that one. – UpQuark Oct 7 at 17:46
  • Feel free to post it as an answer and self-accept. My religion prohibits me from posting SE answers :) – Barry Carter Oct 7 at 21:13
8

Here's some more news APIs you can look into (that are free or cheap - but may have limits on usage):

USA Today API - http://developer.usatoday.com/docs/read/Breaking_News

The Guardian - http://www.theguardian.com/open-platform

The New York Times API - http://developer.nytimes.com/

8

Nearly all newspapers have all kinds of license constraints, making exploitation a legal headache, but there is a welcome exception:

https://en.wikinews.org

All of the Wikinews content is available under the Creative Commons Attribution 2.5 License.

There is an API that lets you get anything.
Unbiased readership statistics are also available.

Wikinews is available in several languages. For some reason, the Serbian edition is particularly popular.

6

R

Your requirements are fairly specific, and I am posting this for future readers of this post having less restrictive requirements.

If you are into R, you can use the tm.plugin.webmining package.

Then you can do things like

googlenews <- WebCorpus(GoogleNewsSource("Microsoft"))
reutersnews <- WebCorpus(ReutersNewsSource("businessNews"))
yahoonews <- WebCorpus(YahooNewsSource("Microsoft"))
nytimes <- WebCorpus(NYTimesSource("Microsoft", appid = "<nytimes_appid>"))

Even if you aren't into R, you can look at the source code and check out how the programmer decided to call the respective APIs.

Date-Range filtering should be possible with the NYT API, but I haven't checked how if the R code can actually do this.

GDELT

Also check out the API of the GDELT project, a huge database which is tracking news+events from around the globe. The API was announced in this blog post, but I haven't used it.

4

If you are looking for generic news sources, and not individual ones (like the NYTimes), then consider CrisisNET. The API works by collecting links to media stories from social media.

(The answer below is modified from here.)

There is a new API for journalists and others called CrisisNET. With it you can search based on location and type. Also, you can use the API and even export CSV files if you aren't familiar with programming.

Explore the API.

example of ebola query

  • It looks down now. Their last blog post is from Sept 2014. – Quentin Aug 31 '17 at 8:52
3

Event Registry

Event registry can provide you the API to answer all the questions you have mentioned. It provides a Python library that can be used to obtained data in JSON format. The library is accessible at https://github.com/gregorleban/EventRegistry/

You are able to search for articles posted in last 3 years (over 100 million) in different languages by over 100.000 news sources.

In addition, it supports also detection of events that are mentioned in these articles. Each event can have several news articles that discuss it and the events can even be linked cross-lingually.

When you register (FREE) you get 1000 free search requests daily.

enter image description here

2

https://webhose.io has some of the attributes you need, but the date filtering may be limited to numbers of days before the current time.

1

Yes. ContextualWeb has a News Search API. Similar to Bing News API and Google News. https://rapidapi.com/contextualwebsearch/api/Web%20Search

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.