Source of multilingual chat/online/contemporary corpus data?

Question

I need to develop autocorrection and prediction for mobile keyboard app that will work for some 8-10 most common Latin languages + Russian (English, German, Spanish, Portugues, French, ...). For start, I need NLP corpus for each of those languages.

Ideally all corpora should be from same source and match at least some of these criteria:

1. Written language, typos removed.
2. Large enough: at least 10 million words, the more the better
3. Texts typed in online chats, emails, forms, apps... Or similar. Should include language typically used in those situations.

I have no need for syntax or other tagging, raw texts (optionally tokenized) are just fine. I can pay reasonable price.

Any help or even a pointer into resource is greatly appreciated!

Answer 1

I think in this case, and because you may need more languages, I'd suggest the Twitter API:

• Filter-Realtime has a filter parameter on language.

This parameter may be used on all streaming endpoints, unless explicitly noted.

Setting this parameter to a comma-separated list of BCP 47 language identifiers corresponding to any of the languages listed on Twitter’s advanced search page will only return Tweets that have been detected as being written in the specified languages. For example, connecting with language=en will only stream Tweets detected to be in the English language.

• Sample Stream endpoint

Returns a small random sample of all public statuses.

---

Benefits to Twitter API:

• Many libraries to easily connect (multiple python, R, etc)

• Structured JSON

• Language field {"lang": "en",} based on automatic detection (which is sometimes not correct, so maybe put a minimum character filter on the tweets).

• May include geolocation or other interesting metadata.

(In the past, there have been better twitter streams, but they've reduced access via the API.)

---

And if the API is not enough data, or too slow, you can find archives of tweets (or just their IDs for API lookup.)

See here for a start.