6

For a tool that I'm working on, I'd like to to distribute an example data set that is Twitter-like. Twitter's terms of service preclude redistributing their data, so I know I can't use them directly. Here is what I need:

  • On the order of tens of thousands of records (if the data set is larger, I can always trim it down).
  • Easily redistributed or linked to, and doesn't require a separate program or agreement for users to access.
  • Includes a timestamp.
  • Includes geo-location as latitude and longitude, or something similar.
  • Includes hashtags or something similar. (Specifically, entries have been tagged/categorized and an individual record may have more than one tag/category.)
  • Preferably, the data topic is something that is of a broad interest, but this isn't a hard requirement.

Note: Twitter technically allows you to kind of redistribute their data, but the method is not sufficient for my purpose. As I understand it, in order to do so you must distribute the IDs of the tweets, and then any user must use their API to get all of the other data about those tweets. This is too complex and time-consuming for what is essentially a demo of the tool.

  • If you have answered this question yourself, in the note, can you add it as an answer and accept it? – philshem May 27 '14 at 12:06
  • @philshem I didn't mean to imply that the note is an answer; it was more to address why redistributing Twitter data won't work for me. I've slightly reworded the note so that hopefully this is clearer. – Tom Panning May 27 '14 at 12:21
  • Maybe statusnet/pump.io/diaspora have some open-license repositories? – Nicolas Raoul May 28 '14 at 5:30
2

stackexchange data has practically all of those requirements:
http://data.stackexchange.com/stackoverflow/query/new

flickr does all of that, although i'm not sure if their use of tags constitutes hashtags, but i'm sure thats easily convertible.
you could also use my personal twitter history that i've downloaded. i could careless. maybe you could get a few more people to download theirs also, combine them into a repo on github?
you could also do the same with my delicious account, although there are no geotags in the set.
i think you could actually apply this to all of the social networks that you can download your personal data from and whose datasets match your requirements. identi.ca comes to mind. although i think they are diaspora now.

  • It is still against the Twitter ToS I believe (IANAL) to distribute tweets or archive them and then share them later. The original poster is correct that only the tweet IDs can be archived. Here is a nice article with background on this issue: inkdroid.org/journal/2014/08/31/on-archiving-tweets – Megan Squire Sep 19 '14 at 17:19
  • good to know....but again, i'll do whatever i please with my twitter data. i highly encourage you to do the same – albert Sep 20 '14 at 21:07
1

Here is a list of folksonomy/social tagging data sets. http://www.markusstrohmaier.info/datasets/ Maybe some of these will help you?

  • Thanks for the link. Unfortunately, it looks like those data sets don't have geo-location information. – Tom Panning Sep 19 '14 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.