1

I have a dataset of tweets annotated by a group of people and I want to distribute it in Github without the user id or Tweet id, do you know if that possible? only the tweet_text and the label.

Given that the work where I collected these tweets was accepted recently in a NLP research conference. Thus, I want to add a link for the dataset in the article.

1

An accurate answer will depend on how many tweets you want to release: MBs, GBs, TBs?

You can take a look at the awesome-twitter-data list to see what other researchers are doing.

One option is to publicly release the tweet_id and user_id and then also a small script for downloading those tweets/users, with instructions for generating and adding your own authentication tokens. This would make more sense for the public (github) and would be fully compliant with the ToS. For the conference, you could additionally post an ephemeral link to the full dataset (text) download, explaining that the link will expire in N months.

Instead of Github, you can also share the data on archive.org, which would not be related to you Github account. See, for example, how ArchiveTeam (not affiliated with archive.org) is storing twitter-stream data: https://archive.org/search.php?query=twitterstream

If you do create a repo with tweet texts, in general, the likely "worst case" would be a takedown notice, which you could comply with if it ever comes to that. If the data is hosted at archive.org, then they could deal with the takedown notice.


Once you've published the dataset, please add it to the list above (and as an answer here!)

| improve this answer | |
  • Thanks @Varvara .. they are just 5k tweets. small dataset. Is it ok? – Ghanem Dec 16 '19 at 9:54
  • For such a small dataset I would say to post it as a single file in a github repo and then also make a copy at archive.org – philshem Dec 16 '19 at 17:23
  • No risks then.? – Ghanem Dec 17 '19 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.