3

I need to crawl research papers, abstracts, titles, and authors and store them for ontological usage. What are the steps to crawl such data?

5

First, you should check for any license for using the site to see if you can use their data for any reason (commercial and non-commercial).

Second, you should read the terms of use and find anything about scraping website. Many websites don't allow scraping.

Third, if non of the above is a problem, you should use a programming language to crawl and scrap the content from the website. Do you have any knowledge of any programming language?

  • Thank you very much, I have a little knowledge about Java or c# but what about web crawler programs such as text crawler or web miner? – Karama Hefny Jul 18 '14 at 19:55
  • 1
    In terms of technical tactics, I highly suggest taking a look at ScraperWiki.com which will host and run your web scraper code and resulting database. You can write scrapers in C#. – Mark Silverberg Jul 18 '14 at 22:30
  • Scrapy would be a good base, or urllib of BeautifulSoup for python. – Dominik Antal Jul 21 '14 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.