1

I am looking for a text data-set. I need this data-set for a set of experiments which compares effectiveness of a set of algorithms working in paragraph-level with the same algorithms working in document-level. For this reason, I want a data-set which has both paragraph-level and document-level labels. It's ok if only a subset of items in each level has labels. I found a lot of paper, which worked on a text processing on both paragraph-level and document-level data-sets but none of their data-sets is publicly available.

Edit: I want to do a set of experiments at the paragraph-level and see if the result is better than learning the same concept at the document-level. As long as labels are binary, it's not important what is labels.

Thank you very much.

  • can you clarify what "at the document level" means? so you want a bunch of content in marked up, <p>, but they're all going to be in an html document (ideally)....so i'm just confused. sorry. – albert Feb 8 '17 at 20:52
2

I think The British National Corpus could do the job. It's a large and free corpus labeled at document, sentence and paragraph level (but also word level, so you'll have a lot of useless xml tags to clean).

It's a big file. The best, for testing, is probably to download the Baby edition (less than 200 Mb once uncompressed). As you can see in the sample below, the documents are labelled <wtext>, the paragraphs <p>, the sentences <s>(and the words <w>, of course).

<bncDoc xml:id="A9Y">
    <teiHeader>
     ...
    </teiHeader>
    <wtext type="NEWS">
        <div level="1" n="13-DEC-89 edition, page 39">
            <head>
                <s n="1">
                    <w c5="NN1" hw="obituary" pos="SUBST">Obituary</w>
                    <c c5="PUN">: </c>
                    <w c5="NP0" hw="dr" pos="SUBST">Dr </w>
                    <w c5="NP0" hw="jack" pos="SUBST">Jack </w>
                    <w c5="NP0" hw="kahn" pos="SUBST">Kahn</w>
                    <c c5="PUN">.</c>
                </s>
            </head>
            <p>
                <s n="1">
                    <w c5="CRD" hw="one" pos="ADJ">ONE </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="NN2" hw="luminary" pos="SUBST">luminaries </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="NN1" hw="child" pos="SUBST">child </w>
                    <w c5="NN1" hw="psychiatry" pos="SUBST">psychiatry </w>
                    <w c5="VHZ" hw="have" pos="VERB">has </w>
                    <w c5="VVN" hw="leave" pos="VERB">left </w>
                    <w c5="PNP" hw="we" pos="PRON">us</w>
                    <c c5="PUN">.</c>
                </s>
                <s n="2">
                    <w c5="NP0" hw="jack" pos="SUBST">Jack </w>
                    <w c5="NP0" hw="kahn" pos="SUBST">Kahn </w>
                    <w c5="VVD" hw="graduate" pos="VERB">graduated </w>
                    <w c5="PRP" hw="with" pos="PREP">with </w>
                    <w c5="NN2" hw="honour" pos="SUBST">honours </w>
                    <w c5="PRP" hw="at" pos="PREP">at </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="NN1" hw="university" pos="SUBST">University </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="NP0" hw="leeds" pos="SUBST">Leeds </w>
                    <w c5="PRP" hw="in" pos="PREP">in </w>
                    <w c5="CRD" hw="1928" pos="ADJ">1928 </w>
                    <w c5="CJC" hw="and" pos="CONJ">and </w>
                    <w c5="VVN-VVD" hw="achieve" pos="VERB">achieved </w>
                    <w c5="AT0" hw="an" pos="ART">an </w>
                    <w c5="NN1" hw="md" pos="SUBST">MD </w>
                    <w c5="PRP" hw="from" pos="PREP">from </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="DT0" hw="same" pos="ADJ">same </w>
                    <w c5="NN1" hw="university" pos="SUBST">university</w>
                    <c c5="PUN">.</c>
                </s>
            </p>
        </div>
    </wtext>
</bncDoc>

Hope this help.

  • Thank you. It's a huge corpus. And I don't understand it's structure. Would you please tell me for example that what I can use for text-level labels and what for paragraph-level labels? – user85361 Feb 8 '17 at 8:00
  • It's good if the labels are not the same in paragraph-level with labels in document-level. – user85361 Feb 8 '17 at 8:11
  • @user85361 I'v edited the answer. – Ettore Rizza Feb 8 '17 at 17:35
  • Thank you. It's very good. What is the labels for paragraphs? Is there any kind of labels for paragraphs or sentences? – user85361 Feb 9 '17 at 10:18
  • What is not clear in the answer ? – Ettore Rizza Feb 10 '17 at 8:18
1

You can also use Text8 corpus of Wikipedia text as mentioned in the research paper Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. I recently worked on a research work and have the dataset you are looking. However you need to wait till 30th March 2017 to have hands on this dataset.

  • Hi, Now, Could you please let have the dataset?Thank you very much. – user85361 Nov 3 '17 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.