I am working on a project in NLP area and in order to test the result I need to compare it against different data sources. I already have NYTimes data source. So, I am looking for another news source data set easily available online. I have heard of Thomson Reuters and WSJs dataset but all through my search I have been unable to find one. Does anyone here know of some dataset?
Are there datasets for scientific journals too? JSTOR, Elsevier anything?
As far as I know, one of the best data sources for NLP has been the ENRON emails. While this is not a "news" source, it would certainly provide some raw data for you to play around with.
Another source is PubMed. While this doesn't provide you with the actual article information, it does provide you with a lot of abstracts for scientific writing. There have even been papers published in PubMed about using this topic.
Finally, there's good old Wikipedia, which allows you to download their entire archived content.
Hope this helps!
Another idea, straying even further from your news focus, would be the movie review sentiment analysis dataset on kaggle: (https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews)
Wikinews.org is part of the Wikimedia project and uses the standard data dump schema (so that the tools for extracting the Wikipedia data dump can be applied to Wikinews).
Search on the index page for enwikinews (en = english)
To get familiar with the content, here is their archive, which provides categories like "region", "topic" and "date".
Event Registry is a system that provides access to over 100 million news articles published in last 3 years in different languages. You can use the provided API (https://github.com/gregorleban/EventRegistry/) to access the content, search by various criteria for articles, events, concepts, etc.
Regarding datasets for scientific journals, the most competitive one that I am aware of is Sci-Hub. It claims to have 48,000,000 scientific academic papers and articles, which is ~90% of all existing papers.
