I need to do some experiments in HBase and Cassandra. To do that I need an adequate dataset. The dataset I'm looking for has to be large enough (more than 2GB) and the data in it has to be sufficiently unstructured to be representative of the kind of problems that relational technology can't cope. Maybe data derived from social networks, and so on. Has anyone that kind of dataset? Or anyone knows where can I find such a dataset? Thanks for your help.
4.5gb twitter data. 41.7 million user profiles, 1.47 billion social relations, 4,262 trending topics, and 106 million tweets. http://an.kaist.ac.kr/traces/WWW2010.html download link bellow: http://an.kaist.ac.kr/~haewoon/release/twitter_social_graph/twitter_rv.zip also you can check site SNAP from Stanford University https://snap.stanford.edu/data/#web or Amazon Public data Sets repository (specially Google or CCAFS-Cimate Data sets) https://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1
you can check also last.fm datasets but are only close to 1.1gb all combined. http://labrosa.ee.columbia.edu/millionsong/lastfm
One more thing: you can get any data and convert to Cassandra and can see difference.
One more repository http://www.cs.waikato.ac.nz/~ml/weka/datasets.html
Ultimately if none of the data fulfill your requirements, than you can generate own data under hadoop cluster (you can get around 10 gb per node, textual data ) and connect to that datasets.