11

What am I looking for ?

I'm looking for a big dataset (at least a few GB) containing French text. This french text should not have too much quality (such as literature or encyclopedias).

Why this specific need ?

I would like to use this dataset to do some sentiment analysis and information extraction in order to measure an entity's reputation. As a consequence, the best dataset could be composed of products reviews, like the well-known amazon review dataset from stanford, or of tweets comments from social networks (please note that I'd rather not produce myself my dataset using rest-like api).

7
  • Are you able or willing to write code?
    – philshem
    Mar 17, 2014 at 14:23
  • I can and have already use twitter's api for getting some informations, if that answers your question. However, I'd rather have an already existing dataset in order to gain some time.
    – merours
    Mar 17, 2014 at 14:24
  • It's very hard to come by twitter datasets because of the ToS. I downloaded 1000+ tweets in 60 seconds with the public stream (4MB with utf-8 encoding), so after 4 hours I would have 240k tweets and around 1GB.
    – philshem
    Mar 17, 2014 at 14:30
  • I see. I'll use the code you provided, then. Regarding the question, I'll wait a little more to see if some interesting datasets are provided.
    – merours
    Mar 17, 2014 at 14:35
  • There are a couple of (commericial?) Amazon APIs that will show up in Google and also the search here. I think it's mostly for getting prices, but you may also find reviews. Bonne chance!
    – philshem
    Mar 17, 2014 at 14:38

7 Answers 7

5

I've been thinking about this question a lot and I have another solution. You correctly wrote that Wikipedia articles have too much quality due to the editing.

But, the discussion (talk) pages are full of raw text data and are not prone to being edited, or even correct.

An example for the French language Open Data page (source):

Y a-t-il une raison pour laquelle le terme Open Data est utilisé partout dans l'article alors qu'il a pour titre Données ouvertes ? Je propose d'être uniforme dans tout l'article et d'opter pour la langue de ce wiki, c'est-à-dire le français. À moins d'une opposition claire, je vais procéder d'ici quelques jours. Dirac (d) 18 février 2013 à 19:48 (CET)

It's straightforward to download all the Discussion/Talk pages not only for articles, but also for Users, Files, Help, etc.

Download:

  • Wikipedia has a (massive) data dump at dumps.wikimedia.org (details). Be careful, the files are huge and won't open with most text viewers. It's best to have a programming approach for parsing the data.

  • The specific files you want are labeleled with something like enwiki-latest-pages-meta-current1.xml-p000000010p000010000.bz2. The key part is pages-meta and then some sequence for the individual files of the split data.

Parse:

  • The header of these XML files contains these (selected) keys:

    <namespace key="1" case="first-letter">Talk</namespace>
    <namespace key="3" case="first-letter">User talk</namespace>
    <namespace key="5" case="first-letter">Wikipedia talk</namespace>
    <namespace key="7" case="first-letter">File talk</namespace>
    <namespace key="9" case="first-letter">MediaWiki talk</namespace>
    <namespace key="11" case="first-letter">Template talk</namespace>
    <namespace key="13" case="first-letter">Help talk</namespace>
    <namespace key="15" case="first-letter">Category talk</namespace>
    <namespace key="101" case="first-letter">Portal talk</namespace>
    <namespace key="109" case="first-letter">Book talk</namespace>
    <namespace key="119" case="first-letter">Draft talk</namespace>
    <namespace key="447" case="first-letter">Education Program talk</namespace>
    <namespace key="711" case="first-letter">TimedText talk</namespace>
    <namespace key="829" case="first-letter">Module talk</namespace>
    
  • Wikimedia's guide to parsing the XML files, a guide to SQL import, and tutorial about how to parse with python: link and source.

3

I can answer for the Twitter data. You can gather plenty of French language tweets by one of two methods:

  • Live Stream. In this case, you can pass this URL: https://stream.twitter.com/1.1/statuses/sample.json?language=fr and you will receive tweets as the standard JSON dictionary. An example python code to download a portion of the public stream is here. In this case, you let the code run and stop it once you have collected a decent sized corpus.

  • Search for French language tweets. To do this programatically, you can use a code like this but replace the search terms with lang:fr. In this case, you search back for a period up to one week for statuses (aka tweets) that are marked as French language.

Notes:

  • For both of these methods, you have to authenticate as a developer first (link).

  • The language is based on Twitter's algorithm for recognizing language - and is not perfect.

  • It's against the Twitter Terms of Service to share raw twitter data, but you can easily collect GBs of data by one the above methods.

3

I cannot help you with products review and you have already got an answer for Twitter. But if you need a dataset with plain text in french, the best solution is the Wikipedia Dump. I used it for another project in English for a similar reason (sentiment analysis and information extraction).

Here is the download link with all the info. The small dataset is 2.6 GB and the full one with the history is more than 20GB.

2
  • 1
    While the data itself is useful, it doesnt quite meet my requirements : texts are too formatted and well written. Also, an article's structure is quite different from a user review. What I'm looking for is more "real" written language, with approximative syntax and vocabulary. I'll edit my question to make it more clear.
    – merours
    Mar 17, 2014 at 15:43
  • @fxm Great! Make it clear that you need something similar to user review to avoid any other non-related answers.
    – Tasos
    Mar 17, 2014 at 15:49
3

How about movies?

Option 1: bulk download of French language movie subtitles. I haven't tried it, but OpenSubtitles.org has both instructions for programming and non-programming.

Option 2: Download plot synopses from the plaintext IMDB database (with reviews) - mirror links. Useful files are 'language.list', 'biographies.list', 'plot.list', etc. Unfortunately I don't see reviews and I checked and only some of the plot summaries for French movies are in French.

There are also some unofficial IMDB API's that may include reviews. You can check them out here and here and here.

2
  • These apis look promising, I'll give them a try. Thanks for all your suggestions, that's some digging you did :D
    – merours
    Mar 18, 2014 at 10:13
  • I frequently have to dig for "diverse" data so I'm used to it.
    – philshem
    Mar 18, 2014 at 10:34
2

You can collect email data in .mbox text files using a search engine. For example, a google search for filetype:mbox from results in plenty of .mbox data. My best attempt at French-only .mbox files is this search.

Individual .mbox files can be concatenated into one large file (linux method):

cat mboxfile1 > mboxfile
echo >> mboxfile
cat mboxfile2 >> mboxfile

To parse the text, one option is the mailbox library in python. See an example code here. Here is the important snippet

import mailbox
mbox = mailbox.mbox('emails.mbox')
for message in mbox:
    print str(message.get_payload()) # print body of each email in emails.mbox file

(my self-source)

Along this line you can also look for French language chat logs (i.e. IRC logs)

2
  • These datas seem difficult to obtain in great quantity, I don't think they suit me.
    – merours
    Mar 18, 2014 at 10:16
  • Yeah, I remember from your other post. I put this answer here for other users that may enjoy mining a mailbox.
    – philshem
    Mar 18, 2014 at 10:31
2

frWaC:

frWaC: a 1.6 billion word corpus constructed from the Web limiting the crawl to the .fr domain and using medium-frequency words from the Le Monde Diplomatique corpus and basic French vocabulary lists as seeds. The corpus was POS-tagged and lemmatized with the TreeTagger, more information available here.

Reference: Baroni, Marco, et al. "The WaCky wide web: a collection of very large linguistically processed web-crawled corpora." Language resources and evaluation 43.3 (2009): 209-226.

0

OpenDataSoft Is a French company that has a lot of French language data.

1
  • I do not see language datasets over there.
    – user4293
    Dec 2, 2016 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.