Historical Twitter Data

Question

I'm trying to get historical twitter data starting from 2010 with specific keywords related to S&P500 companies. I will probably end up using their tickers as keywords once I finalize my list of specific companies.

I'm unable to go back more than a week. Has anyone found a way to get around this? I need to be able to have code available for my research paper/presentation. I've tried TwitterSearch and Tweepy packages.

This is the code using TwitterSearch:

import datetime
from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException

#the start and end of our twitter data
mindate = datetime.date(2010, 01, 01)

maxdate = datetime.date(2016, 01, 01)


try:

    tso = TwitterSearchOrder() # create a TwitterSearchOrder object

    tso.set_keywords(['sp500', 's&p500']) # let's define all words we would like to have a look for
    tso.set_language('en') # English tweets only
    tso.set_include_entities(False) # entities...
    tso.set_until(datetime.date(2016, 01, 01))  #this doesn't work...

    # it's about time to create a TwitterSearch object with our secret tokens
    ts = TwitterSearch(
        consumer_key = 'REMOVED',
        consumer_secret = 'REMOVED',
        access_token = 'REMOVED',
        access_token_secret = 'ALSO REMOVED'
     )

     # this is where the fun actually starts :)
    for tweet in ts.search_tweets_iterable(tso):
        print( '\n@%s tweeted: %s' % \
             ( tweet['user']['screen_name'], tweet['text']) )

except TwitterSearchException as e: 
    print(e)

I also need to be able to print the date of the tweet, how many likes/retweets it had, and possibly the number of followers the user has. Lastly, I need to be able to save all the data in a csv file.

Any help is greatly appreciated!

Answer 1

Unfortunately, the search index for searching tweets only goes back 7 days. Link to the documentation here, it can be found under 'until'.

You might be able to collect tweets from individual users' timelines, if you have an idea about what users are using those terms in their tweets. You can then use the User Timeline part of the REST API to get up to 3200 tweets from each user to see if they have tweeted those terms before?

It's a bit of an indirect method, and you can't tune the API request to only get tweets with your search terms, but it's a potential workaround.

The R equivalent is to use the twitteR package, which comes with methods like getFollowers() which gets a list of followers that follow a particular Twitter account. I'm not sure if there is an equivalent in something like Tweepy.