3

I need to crawl research papers, abstracts, titles, and authors and store them for ontological usage. What are the steps to crawl such data?

Improve this question

1 Answer 1

Reset to default
5

First, you should check for any license for using the site to see if you can use their data for any reason (commercial and non-commercial).

Second, you should read the terms of use and find anything about scraping website. Many websites don't allow scraping.

Third, if non of the above is a problem, you should use a programming language to crawl and scrap the content from the website. Do you have any knowledge of any programming language?

Improve this answer
3
  • Thank you very much, I have a little knowledge about Java or c# but what about web crawler programs such as text crawler or web miner?
    – Karama Hefny
    Jul 18, 2014 at 19:55
  • 1
    In terms of technical tactics, I highly suggest taking a look at ScraperWiki.com which will host and run your web scraper code and resulting database. You can write scrapers in C#.
    – Mark Silverberg
    Jul 18, 2014 at 22:30
  • Scrapy would be a good base, or urllib of BeautifulSoup for python.
    – Dominik Antal
    Jul 21, 2014 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.