I need to crawl research papers, abstracts, titles, and authors and store them for ontological usage. What are the steps to crawl such data?
1 Answer
First, you should check for any license for using the site to see if you can use their data for any reason (commercial and non-commercial).
Second, you should read the terms of use and find anything about scraping website. Many websites don't allow scraping.
Third, if non of the above is a problem, you should use a programming language to crawl and scrap the content from the website. Do you have any knowledge of any programming language?
-
Thank you very much, I have a little knowledge about Java or c# but what about web crawler programs such as text crawler or web miner? Commented Jul 18, 2014 at 19:55
-
1In terms of technical tactics, I highly suggest taking a look at ScraperWiki.com which will host and run your web scraper code and resulting database. You can write scrapers in C#. Commented Jul 18, 2014 at 22:30
-