First of all I would like to apologise for the length of this question and also for my explanation as it might not use the ideal terms.
I am looking for a possible data set that has articles from online sources or otherwise and user comments posted in reply to those articles, similar to a Disqus setup. I would then use this data set to verify that a system I am building is actually identifying the correct areas between two different sets of words. So far I have tried the simple and crude cosine similarity between a comment and a part of the text and the results are not really encouraging as people would use different words to refer to the same object: car-vehicle-automobile etc. (synonyms)
What I am really after is a similarity tagging between a comment and the article or parts of it. For example, if we take this website and all Stack Exchange sites, a user posts a question and other users reply with answers or comments etc. What I would like to have in the data set is a tagging where a particular comment/reply is "linked" to a particular part of the original question. The linking can be numerical etc. It would be similar to what certain API's such as Alchemy output with relevance between a word and the text being processed.
Obviously, the text can be from any source as what is really needed is the "link" between the main text and a short comment. I thought of manual tagging but this takes way too much time. Another option was to ask others to do it (like Mechanical Turk) but then I would not know how accurate that can be and since I would be using this set as my gold standard, I would need to know that it is relatively accurate.
I have been searching high and dry but have not seen anything close to what I would need.
Any help or link to a possible collection would be much appreciated. Thank you.