2

I need to develop autocorrection and prediction for mobile keyboard app that will work for some 8-10 most common Latin languages + Russian (English, German, Spanish, Portugues, French, ...). For start, I need NLP corpus for each of those languages.

Ideally all corpora should be from same source and match at least some of these criteria:

  1. Written language, typos removed.
  2. Large enough: at least 10 million words, the more the better
  3. Texts typed in online chats, emails, forms, apps... Or similar. Should include language typically used in those situations.

I have no need for syntax or other tagging, raw texts (optionally tokenized) are just fine. I can pay reasonable price.

Any help or even a pointer into resource is greatly appreciated!

Improve this question
1
  • Would also help: If there is no such resource known, perhaps there is a way how to create it automatically?
    – Rasto
    Apr 24, 2019 at 21:41

1 Answer 1

Reset to default
2

I think in this case, and because you may need more languages, I'd suggest the Twitter API:

This parameter may be used on all streaming endpoints, unless explicitly noted.

Setting this parameter to a comma-separated list of BCP 47 language identifiers corresponding to any of the languages listed on Twitter’s advanced search page will only return Tweets that have been detected as being written in the specified languages. For example, connecting with language=en will only stream Tweets detected to be in the English language.

Returns a small random sample of all public statuses.

Benefits to Twitter API:

  • Many libraries to easily connect (multiple python, R, etc)

  • Structured JSON

  • Language field {"lang": "en",} based on automatic detection (which is sometimes not correct, so maybe put a minimum character filter on the tweets).

  • May include geolocation or other interesting metadata.

(In the past, there have been better twitter streams, but they've reduced access via the API.)

And if the API is not enough data, or too slow, you can find archives of tweets (or just their IDs for API lookup.)

See here for a start.

Improve this answer
1
  • This is an interesting option - type of texts typed on Twitter is probably quite similar to what is typed generally on mobile... But it will definitelly contain typos so point 1 is not matched which is a problem. I can see ways how to mitigate that issue (combining this source with typo-checked dictionaries) but that also decreases value of the data...
    – Rasto
    May 6, 2019 at 22:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.