1

I am looking for a text data-set. I need this data-set for a set of experiments which compares effectiveness of a set of algorithms working in paragraph-level with the same algorithms working in document-level. For this reason, I want a data-set which has both paragraph-level and document-level labels. It's ok if only a subset of items in each level has labels. I found a lot of paper, which worked on a text processing on both paragraph-level and document-level data-sets but none of their data-sets is publicly available.

Edit: I want to do a set of experiments at the paragraph-level and see if the result is better than learning the same concept at the document-level. As long as labels are binary, it's not important what is labels.

Thank you very much.

Improve this question
1
  • can you clarify what "at the document level" means? so you want a bunch of content in marked up, <p>, but they're all going to be in an html document (ideally)....so i'm just confused. sorry.
    – albert
    Commented Feb 8, 2017 at 20:52

2 Answers 2

Reset to default
2

I think The British National Corpus could do the job. It's a large and free corpus labeled at document, sentence and paragraph level (but also word level, so you'll have a lot of useless xml tags to clean).

It's a big file. The best, for testing, is probably to download the Baby edition (less than 200 Mb once uncompressed). As you can see in the sample below, the documents are labelled <wtext>, the paragraphs <p>, the sentences <s>(and the words <w>, of course). 

<bncDoc xml:id="A9Y">
    <teiHeader>
     ...
    </teiHeader>
    <wtext type="NEWS">
        <div level="1" n="13-DEC-89 edition, page 39">
            <head>
                <s n="1">
                    <w c5="NN1" hw="obituary" pos="SUBST">Obituary</w>
                    <c c5="PUN">: </c>
                    <w c5="NP0" hw="dr" pos="SUBST">Dr </w>
                    <w c5="NP0" hw="jack" pos="SUBST">Jack </w>
                    <w c5="NP0" hw="kahn" pos="SUBST">Kahn</w>
                    <c c5="PUN">.</c>
                </s>
            </head>
            <p>
                <s n="1">
                    <w c5="CRD" hw="one" pos="ADJ">ONE </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="NN2" hw="luminary" pos="SUBST">luminaries </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="NN1" hw="child" pos="SUBST">child </w>
                    <w c5="NN1" hw="psychiatry" pos="SUBST">psychiatry </w>
                    <w c5="VHZ" hw="have" pos="VERB">has </w>
                    <w c5="VVN" hw="leave" pos="VERB">left </w>
                    <w c5="PNP" hw="we" pos="PRON">us</w>
                    <c c5="PUN">.</c>
                </s>
                <s n="2">
                    <w c5="NP0" hw="jack" pos="SUBST">Jack </w>
                    <w c5="NP0" hw="kahn" pos="SUBST">Kahn </w>
                    <w c5="VVD" hw="graduate" pos="VERB">graduated </w>
                    <w c5="PRP" hw="with" pos="PREP">with </w>
                    <w c5="NN2" hw="honour" pos="SUBST">honours </w>
                    <w c5="PRP" hw="at" pos="PREP">at </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="NN1" hw="university" pos="SUBST">University </w>
                    <w c5="PRF" hw="of" pos="PREP">of </w>
                    <w c5="NP0" hw="leeds" pos="SUBST">Leeds </w>
                    <w c5="PRP" hw="in" pos="PREP">in </w>
                    <w c5="CRD" hw="1928" pos="ADJ">1928 </w>
                    <w c5="CJC" hw="and" pos="CONJ">and </w>
                    <w c5="VVN-VVD" hw="achieve" pos="VERB">achieved </w>
                    <w c5="AT0" hw="an" pos="ART">an </w>
                    <w c5="NN1" hw="md" pos="SUBST">MD </w>
                    <w c5="PRP" hw="from" pos="PREP">from </w>
                    <w c5="AT0" hw="the" pos="ART">the </w>
                    <w c5="DT0" hw="same" pos="ADJ">same </w>
                    <w c5="NN1" hw="university" pos="SUBST">university</w>
                    <c c5="PUN">.</c>
                </s>
            </p>
        </div>
    </wtext>
</bncDoc>

Hope this help.

Improve this answer
6
  • Thank you. It's a huge corpus. And I don't understand it's structure. Would you please tell me for example that what I can use for text-level labels and what for paragraph-level labels?
    – user85361
    Commented Feb 8, 2017 at 8:00
  • It's good if the labels are not the same in paragraph-level with labels in document-level.
    – user85361
    Commented Feb 8, 2017 at 8:11
  • @user85361 I'v edited the answer.
    – Ettore Rizza
    Commented Feb 8, 2017 at 17:35
  • Thank you. It's very good. What is the labels for paragraphs? Is there any kind of labels for paragraphs or sentences?
    – user85361
    Commented Feb 9, 2017 at 10:18
  • What is not clear in the answer ?
    – Ettore Rizza
    Commented Feb 10, 2017 at 8:18
1

You can also use Text8 corpus of Wikipedia text as mentioned in the research paper Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. I recently worked on a research work and have the dataset you are looking. However you need to wait till 30th March 2017 to have hands on this dataset.

Improve this answer
1
  • Hi, Now, Could you please let have the dataset?Thank you very much.
    – user85361
    Commented Nov 3, 2017 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.