0

I am using this model for word embeddings trained using word2vec, I want to get the embedding using GloVe to compare the performance. The model is trained using the English Wikipedia corpus, nevertheless, I have not found such a dataset online. Does anyone knows where can I find the dataset?

0

Have a look at this page from the "DBpedia" project which show everything in great detail. DBpedia is a subset of Wikipedia.

Downloadable Files are given in Turtle format (.ttl, compressed as .bz2) which is a plain-text file format.

For more expert advice I would ask around on the Wikidata Telegram group - the Wikipedia and Wikidata developers hang around there, AFAIK.

0

The Wikipedia Corpus - English-Corpora.org

Full-text corpus data

This corpus contains the full text of Wikipedia, and it contains 1.9 billion words in more than 4.4 million articles.

But this corpus allows you to search Wikipedia in a much more powerful way than is possible with the standard interface. You can search by word, phrase, part of speech, and synonyms. You can also find collocates (nearby words), and see re-sortable concordance lines for any word or phrase.

Click on any of the links in the search form to the left for context-sensitive help and see the range of queries the corpus offers.

Most importantly, you can create and use virtual corpora from any of the 4,400,000 articles in the corpus. For example, you could create a corpus with 500-1,000 pages (perhaps 500,000-1,000,000 words) related to microbiology, economics, basketball, Buddhism, or thousands of other topics in less than a minute. (More information, with YouTube videos)

You can then search within that virtual corpus, compare the frequency of a word, phrase, or grammatical construction in your different virtual corpora, and also create "keyword lists" based on the texts in your virtual corpus.

Rather than spending hours or days creating a specialized corpus for a particular topic, you can create it and search it within just a minute or two with this Wikipedia corpus.

Finally, the corpus is related to many other corpora of English that they have created. These corpora were formerly known as the "BYU Corpora", and they offer unparalleled insight into variation in English.


Wikipedia:Database download

Wikipedia offers free copies of all available content to interested users. These databases can be used for mirroring, personal use, informal backups, offline use, or database queries (such as for Wikipedia: Maintenance). All text content is multi-licensed under the Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA) and the GNU Free Documentation License (GFDL). Images and other files are available under different terms, as detailed on their description pages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.