There're two famous n-gram services:
The NY Times Chronicle covers too little (one newspaper), while Google covers too much (all books).
Where can I find n-gram services that covers major mass media alone? Maybe just the press?
LexisNexis had something similar, but it's not open. English corpus datasets I saw lack technical capacity to separate frequencies into sources and years.
If the dataset you want isn't already available, you can create it with some basic programming. Creating an N-gram tool is pretty straightforward, and the volumes of data are not so large if you limit yourself to one topic or a handful of media sources.
Collecting
For this type of project, you are looking to cast a wide net and collect as much data as possible, in order to get a strong signal. Errors in data collection will be lost in the noise (not enough frequency to stand out).
For the data collection, one idea is to use RSS feeds, which come usually in XML format and are easy to parse. For example, for Yahoo News - Politics. You can also use the URL in each short RSS item and then scrape the page.
Another option is the Google News RSS feed or Bing News Developer (details). Here is the URL to an XML output from Google News RSS feed for a particular query.
https://news.google.com/news/feeds?q=apple&output=rss
I like python and I can suggest HTML scraping tools such as lxml, beautifulsoup, scrapy, etc.
You might also look for some news aggregator tools that provide news articles from various sources in a simple text format.
Parsing
After download as much text as possible, it's time to parse. Python's zip module is ready to go (my source):
input_list = ['all', 'this', 'happened', 'more', 'or', 'less']
def find_ngrams(input_list, n):
return zip(*[input_list[i:] for i in range(n)])
By running with n=2, the response is this:
[('all', 'this'), ('this', 'happened'), ('happened', 'more'), ('more', 'or'), ('or', 'less')]
In reality, you need to create input_list by splitting on all punctuation and white space. For example,
import re
text = 'all. this, happened more : or ; less'
print re.split('\W+', text)
would return an array of words from a large piece of text, without punctuation (warning - the '_' character isn't considered punctuation)
There are also some pre-built N-gram tools from NLTK (one, two)
Storage
If the parsing takes some time because of large raw data volumes, I'd write to an the raw N-grams to an intermediate file (probably just CSV). Perhaps you also want to store the source relating the N-gram to the source document (something Google provide), as well as the date (or just year).
Then with some processing you can convert arrays and similar files (like CSV) to a format similar to Google N-grams. To do so, one option is to use the defaultdict library from python's Collections module. To do so, a form of each N-gram would be the key to the dictionary and the integer count would be the value.
from collections import defaultdict
data = defaultdict(int)
for item in find_ngrams(input_list, 2): # using the function from above
data[item] += 1
print data
would return a count for each N-gram (not so exciting in this case).
{('or', 'less'): 1, ('all', 'this'): 1, ('more', 'or'): 1, ('happened', 'more'): 1, ('this', 'happened'): 1})
Further
