7

I'm trying to get historical twitter data starting from 2010 with specific keywords related to S&P500 companies. I will probably end up using their tickers as keywords once I finalize my list of specific companies.

I'm unable to go back more than a week. Has anyone found a way to get around this? I need to be able to have code available for my research paper/presentation. I've tried TwitterSearch and Tweepy packages.

This is the code using TwitterSearch:

import datetime
from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException

#the start and end of our twitter data
mindate = datetime.date(2010, 01, 01)

maxdate = datetime.date(2016, 01, 01)


try:

    tso = TwitterSearchOrder() # create a TwitterSearchOrder object

    tso.set_keywords(['sp500', 's&p500']) # let's define all words we would like to have a look for
    tso.set_language('en') # English tweets only
    tso.set_include_entities(False) # entities...
    tso.set_until(datetime.date(2016, 01, 01))  #this doesn't work...

    # it's about time to create a TwitterSearch object with our secret tokens
    ts = TwitterSearch(
        consumer_key = 'REMOVED',
        consumer_secret = 'REMOVED',
        access_token = 'REMOVED',
        access_token_secret = 'ALSO REMOVED'
     )

     # this is where the fun actually starts :)
    for tweet in ts.search_tweets_iterable(tso):
        print( '\n@%s tweeted: %s' % \
             ( tweet['user']['screen_name'], tweet['text']) )

except TwitterSearchException as e: 
    print(e)

I also need to be able to print the date of the tweet, how many likes/retweets it had, and possibly the number of followers the user has. Lastly, I need to be able to save all the data in a csv file.

Any help is greatly appreciated!

  • I can't find it, but I thought this had come up before ... I think there was a company or two who were archiving twitter and selling access to the historical stuff. – Joe Mar 23 '16 at 21:25
  • I think I found it : opendata.stackexchange.com/a/4056/263 – Joe Mar 23 '16 at 21:26
4

Unfortunately, the search index for searching tweets only goes back 7 days. Link to the documentation here, it can be found under 'until'.

You might be able to collect tweets from individual users' timelines, if you have an idea about what users are using those terms in their tweets. You can then use the User Timeline part of the REST API to get up to 3200 tweets from each user to see if they have tweeted those terms before?

It's a bit of an indirect method, and you can't tune the API request to only get tweets with your search terms, but it's a potential workaround.

The R equivalent is to use the twitteR package, which comes with methods like getFollowers() which gets a list of followers that follow a particular Twitter account. I'm not sure if there is an equivalent in something like Tweepy.

-1

With www.followthehashtag.com twitter analytics service you will be able to get this data, 2010 data is a valuable resource and getting this info will have related costs. They can get all tweets.

There is a sample NASDAQ100 huge dataset here, for free, with all tweets, about 80 days using companies twitter $Symbol (Ticket with an $)

Hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.