Sorry for the somewhat general question, to which the answers necessary depend on the country of operation and necessarily are also subject to become obsolete at some point. But I think it to be quite relevant to the open data initiative, and perhaps on this forum there are people who are informed of this.
Legislation in EU has aimed to form an exemption for mining the data for research purposes. As far as I understand, UK has introduced a rule like this last year. However I am not sure how and where to find up-to-date information on this.
I understand that "the right to read = the right to mine" states that anything that can be seen/read/heard by a human can also be analysed for research purposes. This seems to make sense in a way, as in principle you could physically read all those webpages make notes, collate, and the finish your study. On the other hand doing things automatically can allow things otherwise very difficult or impossible to be manageable, so in this sense the difference can be measured in something else than human time.
Maybe someone here has a good overview of the field now. What are the current state in the EU for research purposed web-crawling with no aimed commercial application? Is the UK law like this in operation?
If the licensing stated on a web-page contradicts this rule, what happens then? The link on UK above seems to imply that the law overrides the web-page licensing (similar to cases of Fair Use in the US I think), and this seems to be exactly the point of the initiative.
Thanks for your help, if you are able to bring some clarity to the issue, or help me phrase the question in a way that can be well answered. Perhaps there are good up-to-date online materials that I should read.