3
votes
Open Source huge dataset for Language Modelling
The Pile
An 800GB Dataset of Diverse Text for Language Modeling
What is the Pile?
The Pile is an 825 GiB diverse, open-source language modeling data set that consists of 22 smaller, high-quality ...
- 1,793
3
votes
Where can I find labelled data for handwritten Chinese characters?
This resource has a link to a bunch of online and off line databases for handwritten letters.
This offline database seems to be a big one, hosted in Japan by Electrotechnical Laboratory has 1.2 ...
3
votes
Accepted
Large English text corpus
Project Gutenberg offers 57.000 free books, available in different formats. Among them, utf-8 encoded plain text with minimal formatting.
The NLTK comes with access to a range of corpora. Among them, ...
- 176
3
votes
Accepted
Looking for datasets of complaints/requests filed at municipalities or cities
US City Open Data Census results for Service Requests should have what you need. And there's always Open 311, though I'm pretty sure its implementers will overlap with the census results.
There is ...
- 11.9k
3
votes
Is there any open data-set for sentence segmentation?
The British National Corpus, "a 100 million word collection of samples of written and spoken language from a wide range of sources", is free of charge and segmented, but i never used it.
- 558
3
votes
Is there any open data-set for sentence segmentation?
Some Corpora in the NLTK have a method sents() that returns each sentence as a list of words, for example the Brown corpus:
import nltk
nltk.download() # this opens a GUI to download all corpora ...
- 406
2
votes
Actual text and corrected text? Or autocorrect data?
OK, this one is a stretch... but instead of comparing text with errors to the corrected text, you can compare original text against incorrect transcriptions.
Reuters Transcribed Subset and data set
...
- 17.7k
2
votes
Accepted
Corpus of English text with tagged locations
There are 1,108,558 English Wikipedia articles that have location information. Location information is specified by Template:Coord in Wikipedia and can be easily extracted from the text.
I guess ...
- 90
2
votes
Public datasets for Text summarization
I have used both PubMed and Clinical Trials datasets for this. The pubmed is by far the larger set of content with over 1M journals.
2
votes
Accepted
Common text data sets in form of panel data
In addition to the answer by @Erwan, I came across
nlp-datasets, collected by Nicolas Iderhoff;
Awesome Public Datasets, collected by AwesomeData.
Both are curated lists of text datasets for NLP ...
- 1,214
2
votes
Common text data sets in form of panel data
It depends what kind of text you're looking for. I can think of the following three sources for text data with years of publication:
Project Gutenberg is a repository containing 60,000 books in the ...
- 121
2
votes
Is there a Simplified Chinese thesaurus file, like Moby for English?
The following tools/ websites seems to do what you are looking for:
usechinese.com/Apps/Thesaurus (also allows for searching using English input)
Chinese-tools.
Chinese Synonyms Thesaurus
However, ...
- 770
2
votes
Is there any text data set with both paragraph and document-level labels available?
I think The British National Corpus could do the job. It's a large and free corpus labeled at document, sentence and paragraph level (but also word level, so you'll have a lot of useless xml tags to ...
- 558
1
vote
Dataset for human-level artificial intelligence
https://en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research list many AI datasets. Note that private companies have released plenty of very useful AI datasets.
- 7,815
1
vote
Index of YouTube video titles
You can use youtube-dl to make queries, and then dump just the metadata as a json, without downloading the video. Then you can parse the (huge) json to get just the ids and titles and categories, or ...
- 17.7k
1
vote
Open Source huge dataset for Language Modelling
https://commoncrawl.org/. Larger (petabytes) but more difficult to use than The Pile. More on the size: https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize
OpenWebText
falcon-refinedweb
- 7,815
1
vote
(Serious) Dataset of paedophilic Youtube comments (or similar)?
Dataset Abusive YouTube Comment
Sexually Abusive Comments and specific words collection from popular youtube videos such as music videos and cartoons (Peppa Pig)
Trending YouTube Video Statistics and ...
- 1,793
1
vote
Known datasets for long document analysis
I'm not an expert on nlp but if your looking for lengthy machine readable corpora you may want to take a look at the HathiTrust project. It's a University lead initiative to make publicly available ...
- 211
1
vote
computer science corpus for language model training
Could the Semantic Scholar Open Research Corpus (S2ORC) be of help?
S2ORC contains 12.7M+ full text papers, of which 7.2% pertain to computer science (according to their paper, at p. 4974).
- 307
1
vote
Corpus suggestion for financial domain
The US Securities and Exchange Commission requires larger publicly traded companies to file quarterly and annual financial statements in XMRL. These can be accessed from the agency's EDGAR site. In ...
- 129
1
vote
Datasets for Topic Modeling
Julian McAuley of UCSD compiles a list of datasets for recommender systems. These are a diverse set of labelled text data fortified with additional measures available for various datasets (e.g. ...
- 1,214
1
vote
Large English text corpus
I have posted a collection of several text datasets related to news here. They contain 50+ mb of text each in an easily parsable format. https://www.kaggle.com/therohk/datasets
- 141
1
vote
Where can I get conversational/directed sentiment data?
I'm afraid you might be asking too much for what is currently available, in particular the requirement that the sentiment is expressed precisely towards the conversational partner. Maybe you can ...
- 156
1
vote
clinical report dataset
Searching for "clinical science reports" (CSR) on Google returns various websites from pharmaceutical companies, which publish summary reports. Maybe these are enough for your research.
As this ...
- 488
1
vote
Is there any text data set with both paragraph and document-level labels available?
You can also use Text8 corpus of Wikipedia text as mentioned in the research paper Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. I recently worked ...
- 59
Only top scored, non community-wiki answers of a minimum length are eligible