The English GigaWord dataset by the Linguistics Dataset Consortium contains some 10 million articles alongside an equally large vocabulary set. It's a lot of organised data. If I had to say it less formally, I'd call it every data scientist's wet dream.

However, it costs a non-member $3000 to obtain.

You heard me. Three grands.

The CNN and DailyMail data that DeepMind uses here sizes up to some 300,000 articles which is a mere 30% which would reflect by a similar proportion in terms of learning.

Might be a long-shot but are there any large open datasets which could be used for the purpose of text summarization?

  • by my calculation it's 3% – philshem Oct 29 '16 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.