I would like to know if there is an existing tool/crawler/scraper/service to crawl the data of a particular user from different websites and applications. For example, can I crawl the data of a user A from website W1, W2 and W3?
Thanks
-
1This is really, really vague, because we don't necessarily know what 'user data' means without knowing the context of the website. For instance, are these all webmail systems? social networking? time & attendance systems? stack exchange sites? – Joe Aug 5 '13 at 16:00
I'm going to assume that, given someone's name, you then want to scrape data from sites that they are posting on or otherwise have information on (i.e. Twitter, Facebook, etc.). With that being assumed...
I have been doing this type of thing for quite some time, and have not found any "out of the box" scrapers that will do this. I've been writing my own using a combination of Ruby and Python, which ultimately may be the way you have to go.
Note: be sure to check the TOS of each site, especially social sites. Some don't like their data going outside of their private walls.
A Google search will come up with a few tools that require you to learn XPATH or Regex. Both are relatively simple, once you get the hang of it (which took me some time).
If you clarify what type of information you're looking for I'll be able to give you a better answer.
-
Thanks for the answer. Basically I am interested in knowing a user's interest in different domains. For example, the books he likes (lets say from amazon), the places of he interet (lets say booking.com) and his taste in music (may be deezer or amazon). – user156379 Aug 6 '13 at 7:12
-
To add to this discussion, I found this paper "An Unsupervised Approach to Discovering and Disambiguating Social Media Profiles" very useful. Thanks – user156379 Aug 7 '13 at 8:57
