6

For a tool that I'm working on, I'd like to to distribute an example data set that is Twitter-like. Twitter's terms of service preclude redistributing their data, so I know I can't use them directly. Here is what I need:

  • On the order of tens of thousands of records (if the data set is larger, I can always trim it down).
  • Easily redistributed or linked to, and doesn't require a separate program or agreement for users to access.
  • Includes a timestamp.
  • Includes geo-location as latitude and longitude, or something similar.
  • Includes hashtags or something similar. (Specifically, entries have been tagged/categorized and an individual record may have more than one tag/category.)
  • Preferably, the data topic is something that is of a broad interest, but this isn't a hard requirement.

Note: Twitter technically allows you to kind of redistribute their data, but the method is not sufficient for my purpose. As I understand it, in order to do so you must distribute the IDs of the tweets, and then any user must use their API to get all of the other data about those tweets. This is too complex and time-consuming for what is essentially a demo of the tool.

Improve this question
2
  • @philshem I didn't mean to imply that the note is an answer; it was more to address why redistributing Twitter data won't work for me. I've slightly reworded the note so that hopefully this is clearer.
    – Tom Panning
    Commented May 27, 2014 at 12:21
  • Maybe statusnet/pump.io/diaspora have some open-license repositories?
    – Nicolas Raoul ♦
    Commented May 28, 2014 at 5:30

2 Answers 2

Reset to default
2

stackexchange data has practically all of those requirements:
http://data.stackexchange.com/stackoverflow/query/new

flickr does all of that, although i'm not sure if their use of tags constitutes hashtags, but i'm sure thats easily convertible.
you could also use my personal twitter history that i've downloaded. i could careless. maybe you could get a few more people to download theirs also, combine them into a repo on github?
you could also do the same with my delicious account, although there are no geotags in the set.
i think you could actually apply this to all of the social networks that you can download your personal data from and whose datasets match your requirements. identi.ca comes to mind. although i think they are diaspora now.

Improve this answer
2
  • It is still against the Twitter ToS I believe (IANAL) to distribute tweets or archive them and then share them later. The original poster is correct that only the tweet IDs can be archived. Here is a nice article with background on this issue: inkdroid.org/journal/2014/08/31/on-archiving-tweets
    – Megan Squire
    Commented Sep 19, 2014 at 17:19
  • good to know....but again, i'll do whatever i please with my twitter data. i highly encourage you to do the same
    – albert
    Commented Sep 20, 2014 at 21:07
1

Here is a list of folksonomy/social tagging data sets. http://www.markusstrohmaier.info/datasets/ Maybe some of these will help you?

Improve this answer
1
  • Thanks for the link. Unfortunately, it looks like those data sets don't have geo-location information.
    – Tom Panning
    Commented Sep 19, 2014 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.