2

I'll be delving into text mining applications for my master thesis and I need data for it. Ideally, I would need a corpora of texts/news articles from some single (or multiple) credible and authoritative source covering financial markets/economy and the like, spanning a time period as long as possible.

It turns out that there is data somewhat what I'm looking for: the Wall Street Journal corpus or Reuters corpus. The problem is that these datasets have many texts for each day, but spanning only few years time.
For my purposes, I would be more interested to have only a few texts/news articles for each day/workday but the whole corpora spanning a decade or more.

Does anyone know of datasets/corporas that would suit my requirements?
Any other possibilities/ideas to construct one such dataset using open resources?

migrated from datascience.stackexchange.com Jul 15 '15 at 11:00

This question came from our site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field.

  • The problem in my eyes is survivorship bias. Many many companies from the early 2000s don't exist anymore or have been merged with other companies. Do you think it's worth digitising old paper newspapers for the sake of it? – hroptatyr Jul 16 '15 at 15:48
  • Other than that, try Bloomberg News. While not open, they have an archive that spans at least 13 years. – hroptatyr Jul 16 '15 at 15:50
  • @hroptatyr thats a valid point. I haven't thought about it. though, mostly I guess I am concerned with sources that are already digital, such as bloomberg news. I just have no idea how/what/who I should approach with an inquiry for such resources? – Sarunas Jul 19 '15 at 16:15
3

Your school's library might have access to databases of newspaper and magazine archives, some going back decades. If you're in a large city, your public library might also have those. The New York Times is one of the most common, but some databases, like ProQuest, have many different archived titles.

To get the articles over decades, you may need to download more than you need and apply some criteria to select only a few per day. The criteria, of course, would depend on your project.

3

Check out those datasets here:

8.5M titles from Reuters (2007-2016)

https://github.com/philipperemy/Reuters-full-data-set

450k + 100k news from Bloomberg / Reuters (more than a decade)

https://github.com/philipperemy/financial-news-dataset

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.