Large French dataset for NLP (not formal, rather discussions/reviews)

Question

What am I looking for ?

I'm looking for a big dataset (at least a few GB) containing French text. This french text should not have too much quality (such as literature or encyclopedias).

Why this specific need ?

I would like to use this dataset to do some sentiment analysis and information extraction in order to measure an entity's reputation. As a consequence, the best dataset could be composed of products reviews, like the well-known amazon review dataset from stanford, or of tweets comments from social networks (please note that I'd rather not produce myself my dataset using rest-like api).

Answer 1

I've been thinking about this question a lot and I have another solution. You correctly wrote that Wikipedia articles have too much quality due to the editing.

But, the discussion (talk) pages are full of raw text data and are not prone to being edited, or even correct.

An example for the French language Open Data page (source):

Y a-t-il une raison pour laquelle le terme Open Data est utilisé partout dans l'article alors qu'il a pour titre Données ouvertes ? Je propose d'être uniforme dans tout l'article et d'opter pour la langue de ce wiki, c'est-à-dire le français. À moins d'une opposition claire, je vais procéder d'ici quelques jours. Dirac (d) 18 février 2013 à 19:48 (CET)

It's straightforward to download all the Discussion/Talk pages not only for articles, but also for Users, Files, Help, etc.

Download:

• Wikipedia has a (massive) data dump at dumps.wikimedia.org (details). Be careful, the files are huge and won't open with most text viewers. It's best to have a programming approach for parsing the data.

  • What's available?
  • English latest, French latest.

• The specific files you want are labeleled with something like enwiki-latest-pages-meta-current1.xml-p000000010p000010000.bz2. The key part is pages-meta and then some sequence for the individual files of the split data.

Parse:

• The header of these XML files contains these (selected) keys:

  <namespace key="1" case="first-letter">Talk</namespace>
  <namespace key="3" case="first-letter">User talk</namespace>
  <namespace key="5" case="first-letter">Wikipedia talk</namespace>
  <namespace key="7" case="first-letter">File talk</namespace>
  <namespace key="9" case="first-letter">MediaWiki talk</namespace>
  <namespace key="11" case="first-letter">Template talk</namespace>
  <namespace key="13" case="first-letter">Help talk</namespace>
  <namespace key="15" case="first-letter">Category talk</namespace>
  <namespace key="101" case="first-letter">Portal talk</namespace>
  <namespace key="109" case="first-letter">Book talk</namespace>
  <namespace key="119" case="first-letter">Draft talk</namespace>
  <namespace key="447" case="first-letter">Education Program talk</namespace>
  <namespace key="711" case="first-letter">TimedText talk</namespace>
  <namespace key="829" case="first-letter">Module talk</namespace>
  

• Wikimedia's guide to parsing the XML files, a guide to SQL import, and tutorial about how to parse with python: link and source.

Answer 2

I can answer for the Twitter data. You can gather plenty of French language tweets by one of two methods:

• Live Stream. In this case, you can pass this URL: https://stream.twitter.com/1.1/statuses/sample.json?language=fr and you will receive tweets as the standard JSON dictionary. An example python code to download a portion of the public stream is here. In this case, you let the code run and stop it once you have collected a decent sized corpus.

• Search for French language tweets. To do this programatically, you can use a code like this but replace the search terms with lang:fr. In this case, you search back for a period up to one week for statuses (aka tweets) that are marked as French language.

Notes:

• For both of these methods, you have to authenticate as a developer first (link).

• The language is based on Twitter's algorithm for recognizing language - and is not perfect.

• It's against the Twitter Terms of Service to share raw twitter data, but you can easily collect GBs of data by one the above methods.

Answer 3

I cannot help you with products review and you have already got an answer for Twitter. But if you need a dataset with plain text in french, the best solution is the Wikipedia Dump. I used it for another project in English for a similar reason (sentiment analysis and information extraction).

Here is the download link with all the info. The small dataset is 2.6 GB and the full one with the history is more than 20GB.

Answer 4

How about movies?

Option 1: bulk download of French language movie subtitles. I haven't tried it, but OpenSubtitles.org has both instructions for programming and non-programming.

Option 2: Download plot synopses from the plaintext IMDB database (with reviews) - mirror links. Useful files are 'language.list', 'biographies.list', 'plot.list', etc. Unfortunately I don't see reviews and I checked and only some of the plot summaries for French movies are in French.

There are also some unofficial IMDB API's that may include reviews. You can check them out here and here and here.

Answer 5

You can collect email data in .mbox text files using a search engine. For example, a google search for filetype:mbox from results in plenty of .mbox data. My best attempt at French-only .mbox files is this search.

Individual .mbox files can be concatenated into one large file (linux method):

cat mboxfile1 > mboxfile
echo >> mboxfile
cat mboxfile2 >> mboxfile

To parse the text, one option is the mailbox library in python. See an example code here. Here is the important snippet

import mailbox
mbox = mailbox.mbox('emails.mbox')
for message in mbox:
    print str(message.get_payload()) # print body of each email in emails.mbox file

(my self-source)

Along this line you can also look for French language chat logs (i.e. IRC logs)

Answer 6

frWaC:

frWaC: a 1.6 billion word corpus constructed from the Web limiting the crawl to the .fr domain and using medium-frequency words from the Le Monde Diplomatique corpus and basic French vocabulary lists as seeds. The corpus was POS-tagged and lemmatized with the TreeTagger, more information available here.

Reference: Baroni, Marco, et al. "The WaCky wide web: a collection of very large linguistically processed web-crawled corpora." Language resources and evaluation 43.3 (2009): 209-226.

Answer 7

OpenDataSoft Is a French company that has a lot of French language data.