26

Where can I get Twitter datasets available for analysis?

I found two:

Improve this question
4
  • 2
    Can you specify what kind of data do you want? For instance, number of followers and following? Relationships? Tweets? Retweets? etc
    – Tasos
    Commented Feb 15, 2014 at 14:39
  • 1
    Yes, please clarify -- I have a long list of Twitter screen names for Apache Developers, but I don't know if you want tweet or names.
    – Megan Squire
    Commented Feb 15, 2014 at 17:41
  • Preferably tweets as they're returned by Twitter's API (all fields). API itself is too restrictive for analysis. But other data would be useful as well.
    – Anton Tarasenko
    Commented Feb 16, 2014 at 1:32
  • You might also find the answer to a similar question helpful in looking for historical information: opendata.stackexchange.com/questions/4006/…
    – Jeanne Holm
    Commented Nov 16, 2014 at 13:08

7 Answers 7

Reset to default
14

While you will find some exceptions, sharing archived tweets is against Twitter Developer Policy.

Take all reasonable efforts to do the following, provided that when requested by Twitter, you must promptly take such actions:

  • Delete Content that Twitter reports as deleted or expired;
  • etc

If a user deletes their own content, the archive should reflect that deletion (which is a massive effort to continuously check).

For that reason, Twitter data sets are often shared as simply two fields: user_id and tweet_id. Then to reconstruct the dataset, one would query the API with those two keys. Then Twitter can ensure that if the tweet was deleted after the initial grab, the content won't show up in the second.

As an exception... the banning of Politwoops, a service from Sunlight Foundation that archive deleted tweets from politicians. It has since been reinstated.

It doesn’t appear that Twitter is changing its terms for developers. Instead, the company has reached an agreement with the Sunlight Foundation and the Open State Foundation, another group involved in versions of Politwoops around the world, to offer the an exception.

If you run the live stream, you can collect your own (you'll get more tweets than you can deal with).

edit: I can add that if you want to 'search' for all tweets, you can use 'lang:en' as your search parameter. See here. Then you can loop over desired langauges.

Improve this answer
7

72 hours #gamergate scrape provides a link to download the dataset.

Improve this answer
2
  • 2
    Welcome to OpenData.SX! Your linked dataset looks great, but could you please extend your answer a bit to make it clear what the dataset is about?
    – Patrick Hoefler
    Commented Oct 28, 2014 at 21:12
  • 4
    The author of the article has taken down the dataset, but you can still access the 9mb CSV file of Tweet IDs: waxy.org/random/misc/gamergate_tweets.csv
    – philshem
    Commented Oct 29, 2014 at 7:43
7

scraperwiki lets you search twitter for terms, but also allows twitters search operators. here's a quick search for my handle:
https://scraperwiki.com/dataset/crt52ai/view/fpzct4y
they also offer a get a list of followers feature, so not exactly what you are looking for, but a twitter tool nonetheless

EDIT: some sets i came across:
Twitter Data used for sentiment analysis
http://help.sentiment140.com/for-students/
network Twitter Data
http://snap.stanford.edu/data/higgs-twitter.html
social Twitter data
http://snap.stanford.edu/data/egonets-Twitter.html

EDIT 2: Three more sets:
Arizona State University Twitter Data Set
http://academictorrents.com/details/2399616d26eeb4ae9ac3d05c7fdd98958299efa9
Twitter 2010-10 Dataset
http://academictorrents.com/details/d8b3a315172c8d804528762f37fa67db14577cdb
Twitter Data - NIPS 2012
http://academictorrents.com/details/046cf7a75db2a530b1505a4ce125fbe0031f4661

Improve this answer
5

Check out these links below, and you can also search "twitter" in data.world for other related data. There's quite a few users who are already aggregating this data and creating analyses and cool visualizations. If you're interested in collaborating, some are seeking additional contributors to help build out these datasets further. Hope this helps!

https://data.world/dash/twitter-transparency-reports

https://data.world/socialmediadata/social-circles-from-twitter

Improve this answer
4

There are a several ways to get access to historical Twitter data. Here are some options:

  • I've had great luck using Topsy in looking at a wide variety of tweets ranging from disease vectors in Africa to sentiment analysis. Here's the link for 28K #opendata tweets for the last 20 days. You can expand to "all time", search by language, and look at influencers.
  • Use the Twitter API to get the data you can for free. Good developer resources are available.
  • The most comprehensive historical archive may be via Gnip, but unfortunately it is not free and it's unclear what the actual costs are.
Improve this answer
3

Here is a dataset of relative word frequency for the top 10,000 words in 890 million Tweets, divided by county: https://sites.google.com/site/wordmapperinfo/

Improve this answer
2
  • This gives word frequency lists, not tweets.
    – user4293
    Commented Jan 27, 2017 at 9:24
  • 1
    True, but OP didn't specify what kind of Twitter data he or she was looking for.
    – wirefire
    Commented Jan 30, 2017 at 16:17
0

Just discovered the awesome-twitter-data list

https://github.com/shaypal5/awesome-twitter-data

A list of Twitter datasets and related resources, released under CC0.

Table of Contents, as of now()

1   Twitter Datasets

    1.1   Tweet datasets
        1.1.1   Tweet ID datasets
    1.2   Tweet datasets (labelled)
    1.3   User datasets
    1.4   Lost Datasets

2   Other Lists
3   Tools

    3.1   Data Collection
    3.2   Analysis

4   Academic Papers

    4.1   Demographics Prediction

5   Articles & blog posts

Tweet Datasets

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.