5

I'm not aware of an available dataset in the requested form, but it should be possible to obtain something similar by parsing data from wiktionary. The latest dump can be obtained at https://dumps.wikimedia.org/enwiktionary/latest/ Also, some interesting links can be found by googling "wiktionary rdf"


2

If you want to find which words are most common you can scan 1 billion web pages from 2009 in 10 languages. It's 5TB compressed. They have subsets of data too. Common Crawl web pages are from Oct 2015 and previous years. More web page collections here at Rockstar datasets on Github.


2

The criteria for grouping "semantically equivalent licenses" might be implemented as an external HTTP REST service. There are already some services based on that dataset running here: http://licensius.com/apidoc/index.html Volume of licenses. Contributions of new licenses or suggested revisions to this github folder are welcome; only a TTL has to be dropped ...


2

It seems there are some commercial databases, but if you are brave and don't need an exhaustive list for 100% coverage, you could use Wikipedia articles for Arabic people top level : List_of_Arabic_given_names mid level : Abeer bottom level : Abeer_Hamza Other top levels : Category:Arabic_feminine_given_names, Category:Arabic-language_names Wikipedia ...


2

Wikidata have some place name but those are only for notable locations Japanese government have a Gazetter of Japan which contain translation of ~4K place name in Japan http://www.gsi.go.jp/kihonjohochousa/gazetteer.html Japanese government have also published a guide that indicate how to romanize Japanese place names. http://www.gsi.go.jp/common/000111876....


1

ENAMDICT/JMnedict is a database of Japanese proper names: http://www.edrdg.org/enamdict/enamdict_doc.html It also contains other types of proper names, but place/shops/building names are those marked with either p, c, o, st. License: CC-BY-SA 3.0 http://www.edrdg.org/edrdg/licence.html not sure whether it is compatible with OSM. Drawback: It only ...


1

Check out OPUS, an open parallel corpus. OPUS is a growing collection of translated texts from the web. In the OPUS project we try to convert and align free online data, to add linguistic annotation, and to provide the community with a publicly available parallel corpus. OPUS is based on open source products and the corpus is also delivered as an open ...


1

The term to search for is "parallel corpus". The most comprehensive resource for Swedish/English seems to be: The English-Swedish Parallel Corpus (ESPC) The current (expanded) version of the ESPC consists of 64 English text samples and their translations into Swedish and 72 Swedish text samples and their translations into English (see Table 1). ...


1

You might have found that commercial one already and it is not multi-lingual but would cover all your use cases (you would have to translate your ingredient names to English using Google Translate though) https://spoonacular.com/food-api


1

There are probabilities for English with all EU languages (except Irish and Croatian) available here. Inside the .zip for each language pair English-X you will find files lex.e2x and lex.x2e. They contain strictly unigram-to-unigram probabilities, which maintains simplicity but is an inherent limitation. The files are the result of a 2014 paper by Aker, ...


1

I know someone made a converter http://www.sljfaq.org/cgi/k2e.cgi that convert kana into possible foreign word, but it is a multi match system that each kana are given multiple possible corresponding answer. It probably doesn't match your need due to the too many outcome and is not open, but might be it's worth to further investigate about it. Contact is on ...


1

Such a dataset was used to create the BLEU metric, but I do not know if it is available. Maybe you could try asking the authors of the paper that introduced it if they could share their data? Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation. ACL-2002: 40th Annual meeting of the ...


Only top voted, non community-wiki answers of a minimum length are eligible