0

I would like to make an auto-completion tool for people who are don't type very well, like what exists on phones. So I do not need just a dictionary but a list of all words, like all conjugations of verbs and all word variants (plural, gender). I can use use the wiktionary's word frequency list, but it is not as complete as I would like. Yet, the frequency sorting property is interesting for me.

So should I look for something else or should I start with what I have?

2

If you really just need all possible spellings, then I would suggest downloading Google N-Grams for N=1 (single words). These are individual words/strings coming from books in the French language. You also have the year, the count in that book, and the part of speech, if available. From this data you can create frequency lists of all unique spellings.

http://storage.googleapis.com/books/ngrams/books/datasetsv2.html

enter image description here

The individual files are quite large. For example, a single file, googlebooks-fre-all-1gram-20120701-a.gz is 194 MB compressed, and 1.0 GB uncompressed. All full of single words.

The structure is quite simple

ngram TAB year TAB match_count TAB volume_count NEWLINE

So an actual line in the 1-gram files will look like this:

âutre_ADJ   1794    1   1

where _ADJ defines the part of speech.

License is Creative Commons Attribution 3.0 Unported.


p.s. I'm a big fan of Python's Collections library for the task of counting frequency of something too large to read at once. See here for example of counting characters. There is also a Python N-gram downloader package, but in this case I prefer to get the files manually (since they are needed to be downloaded only once).

  • this dataset is very interesting. But too big for me. Sources seems also to not correspond with an everyday language. – Gael May 25 '17 at 18:07
1

I'm no expert on this topic, but I believe that I have suggestion to make.

Start with the wiktionary word frequency list that is "not as complete as you would like," because it is still your best starting point.

Then find and fix the (occasional) "holes" after the fact.

  • It is what I have done, but this list was not as complete as it has to make my tool useful. – Gael May 25 '17 at 18:09
0

I have found the perfect dataset for me: http://www.lexique.org/

Word are extracted from books and movies subtitles. It contains about 140,000 words.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.