Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
3 votes
Accepted

Large English text corpus

Project Gutenberg offers 57.000 free books, available in different formats. Among them, utf-8 encoded plain text with minimal formatting. The NLTK comes with access to a range of corpora. Among them, ...
Lutz Büch's user avatar
3 votes

Where can I find labelled data for handwritten Chinese characters?

This resource has a link to a bunch of online and off line databases for handwritten letters. This offline database seems to be a big one, hosted in Japan by Electrotechnical Laboratory has 1.2 ...
Aksakal almost surely binary's user avatar
3 votes

Open Source huge dataset for Language Modelling

The Pile An 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is an 825 GiB diverse, open-source language modeling data set that consists of 22 smaller, high-quality ...
Orophile's user avatar
  • 1,761
3 votes
Accepted

Looking for datasets of complaints/requests filed at municipalities or cities

US City Open Data Census results for Service Requests should have what you need. And there's always Open 311, though I'm pretty sure its implementers will overlap with the census results. There is ...
albert's user avatar
  • 11.9k
3 votes

Is there any open data-set for sentence segmentation?

The British National Corpus, "a 100 million word collection of samples of written and spoken language from a wide range of sources", is free of charge and segmented, but i never used it.
Ettore Rizza's user avatar
3 votes

Is there any open data-set for sentence segmentation?

Some Corpora in the NLTK have a method sents() that returns each sentence as a list of words, for example the Brown corpus: import nltk nltk.download() # this opens a GUI to download all corpora ...
Suzana's user avatar
  • 406
2 votes

Actual text and corrected text? Or autocorrect data?

OK, this one is a stretch... but instead of comparing text with errors to the corrected text, you can compare original text against incorrect transcriptions. Reuters Transcribed Subset and data set ...
philshem's user avatar
  • 17.6k
2 votes

Formats of texts from wikisource?

If you are just interested in two dozen text I think it is easiest just to request for the raw wikitext, for instance for the 'The Metal Pig' page: https://en.wikisource.org/wiki/The_Metal_Pig?action=...
Finn Årup Nielsen's user avatar
2 votes

List of political blogs

There is a blog that lists political blogs based in the US: http://politicalbloglistings.blogspot.ch/ Categories (count of listed blogs): Conservative (108) Liberal (70) Non/Bi-Partisan or ...
philshem's user avatar
  • 17.6k
2 votes
Accepted

Corpus of English text with tagged locations

There are 1,108,558 English Wikipedia articles that have location information. Location information is specified by Template:Coord in Wikipedia and can be easily extracted from the text. I guess ...
Taha's user avatar
  • 90
2 votes

Public datasets for Text summarization

I have used both PubMed and Clinical Trials datasets for this. The pubmed is by far the larger set of content with over 1M journals.
Liam Cavanagh - MSFT's user avatar
2 votes

Is there a Simplified Chinese thesaurus file, like Moby for English?

The following tools/ websites seems to do what you are looking for: usechinese.com/Apps/Thesaurus (also allows for searching using English input) Chinese-tools. Chinese Synonyms Thesaurus However, ...
SuperKogito's user avatar
2 votes

Is there any text data set with both paragraph and document-level labels available?

I think The British National Corpus could do the job. It's a large and free corpus labeled at document, sentence and paragraph level (but also word level, so you'll have a lot of useless xml tags to ...
Ettore Rizza's user avatar
2 votes
Accepted

Common text data sets in form of panel data

In addition to the answer by @Erwan, I came across nlp-datasets, collected by Nicolas Iderhoff; Awesome Public Datasets, collected by AwesomeData. Both are curated lists of text datasets for NLP ...
sboysel's user avatar
  • 1,214
2 votes

Common text data sets in form of panel data

It depends what kind of text you're looking for. I can think of the following three sources for text data with years of publication: Project Gutenberg is a repository containing 60,000 books in the ...
Erwan's user avatar
  • 121
1 vote

Index of YouTube video titles

You can use youtube-dl to make queries, and then dump just the metadata as a json, without downloading the video. Then you can parse the (huge) json to get just the ids and titles and categories, or ...
philshem's user avatar
  • 17.6k
1 vote

Open Source huge dataset for Language Modelling

https://commoncrawl.org/. Larger (petabytes) but more difficult to use than The Pile. More on the size: https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize OpenWebText falcon-refinedweb
Franck Dernoncourt's user avatar
1 vote

(Serious) Dataset of paedophilic Youtube comments (or similar)?

Dataset Abusive YouTube Comment Sexually Abusive Comments and specific words collection from popular youtube videos such as music videos and cartoons (Peppa Pig) Trending YouTube Video Statistics and ...
Orophile's user avatar
  • 1,761
1 vote

Known datasets for long document analysis

I'm not an expert on nlp but if your looking for lengthy machine readable corpora you may want to take a look at the HathiTrust project. It's a University lead initiative to make publicly available ...
Matt's user avatar
  • 211
1 vote

computer science corpus for language model training

Could the Semantic Scholar Open Research Corpus (S2ORC) be of help? S2ORC contains 12.7M+ full text papers, of which 7.2% pertain to computer science (according to their paper, at p. 4974).
anpami's user avatar
  • 307
1 vote

Corpus suggestion for financial domain

The US Securities and Exchange Commission requires larger publicly traded companies to file quarterly and annual financial statements in XMRL. These can be accessed from the agency's EDGAR site. In ...
Richard Careaga's user avatar
1 vote

Datasets for Topic Modeling

Julian McAuley of UCSD compiles a list of datasets for recommender systems. These are a diverse set of labelled text data fortified with additional measures available for various datasets (e.g. ...
sboysel's user avatar
  • 1,214
1 vote

Large English text corpus

I have posted a collection of several text datasets related to news here. They contain 50+ mb of text each in an easily parsable format. https://www.kaggle.com/therohk/datasets
Rohit's user avatar
  • 141
1 vote

Where can I get conversational/directed sentiment data?

I'm afraid you might be asking too much for what is currently available, in particular the requirement that the sentiment is expressed precisely towards the conversational partner. Maybe you can ...
mapto's user avatar
  • 156
1 vote

clinical report dataset

Searching for "clinical science reports" (CSR) on Google returns various websites from pharmaceutical companies, which publish summary reports. Maybe these are enough for your research. As this ...
Grimaldi's user avatar
  • 488
1 vote

Is there any text data set with both paragraph and document-level labels available?

You can also use Text8 corpus of Wikipedia text as mentioned in the research paper Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. I recently worked ...
cpwah's user avatar
  • 59

Only top scored, non community-wiki answers of a minimum length are eligible