8

I'm trying to get historical twitter data starting from 2010 with specific keywords related to S&P500 companies. I will probably end up using their tickers as keywords once I finalize my list of specific companies.

I'm unable to go back more than a week. Has anyone found a way to get around this? I need to be able to have code available for my research paper/presentation. I've tried TwitterSearch and Tweepy packages.

This is the code using TwitterSearch:

import datetime
from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException

#the start and end of our twitter data
mindate = datetime.date(2010, 01, 01)

maxdate = datetime.date(2016, 01, 01)


try:

    tso = TwitterSearchOrder() # create a TwitterSearchOrder object

    tso.set_keywords(['sp500', 's&p500']) # let's define all words we would like to have a look for
    tso.set_language('en') # English tweets only
    tso.set_include_entities(False) # entities...
    tso.set_until(datetime.date(2016, 01, 01))  #this doesn't work...

    # it's about time to create a TwitterSearch object with our secret tokens
    ts = TwitterSearch(
        consumer_key = 'REMOVED',
        consumer_secret = 'REMOVED',
        access_token = 'REMOVED',
        access_token_secret = 'ALSO REMOVED'
     )

     # this is where the fun actually starts :)
    for tweet in ts.search_tweets_iterable(tso):
        print( '\n@%s tweeted: %s' % \
             ( tweet['user']['screen_name'], tweet['text']) )

except TwitterSearchException as e: 
    print(e)

I also need to be able to print the date of the tweet, how many likes/retweets it had, and possibly the number of followers the user has. Lastly, I need to be able to save all the data in a csv file.

Any help is greatly appreciated!

  • I can't find it, but I thought this had come up before ... I think there was a company or two who were archiving twitter and selling access to the historical stuff. – Joe Mar 23 '16 at 21:25
  • I think I found it : opendata.stackexchange.com/a/4056/263 – Joe Mar 23 '16 at 21:26
4

Unfortunately, the search index for searching tweets only goes back 7 days. Link to the documentation here, it can be found under 'until'.

You might be able to collect tweets from individual users' timelines, if you have an idea about what users are using those terms in their tweets. You can then use the User Timeline part of the REST API to get up to 3200 tweets from each user to see if they have tweeted those terms before?

It's a bit of an indirect method, and you can't tune the API request to only get tweets with your search terms, but it's a potential workaround.

The R equivalent is to use the twitteR package, which comes with methods like getFollowers() which gets a list of followers that follow a particular Twitter account. I'm not sure if there is an equivalent in something like Tweepy.

0

I will suggest you to use Trackmyhashtag. I personally prefer this tool to extract historical data of my important hashtag and keywords.

With the help of this tool, you can get historical Twitter data of any hashtag, keyword or account for any time period since 2006.

It can provide you what exactly you need. With complete analysis data it provides an online dashboard with lots of metrics, list of all the tweets with content, time & date, likes, impressions, retweets, complete user detail who have tweeted by using that specific keyword or hashtag. To get Twitter data, visit Trackmyhashtag website and click on "Historical data'' - scroll down and you will see a request data. Input your search term and select the dates. Within an hour, you will receive your data and other details with it. I hope I helped!

New contributor
Caitlyn Davis is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
-2

With www.followthehashtag.com twitter analytics service you will be able to get this data, 2010 data is a valuable resource and getting this info will have related costs. They can get all tweets.

There is a sample NASDAQ100 huge dataset here, for free, with all tweets, about 80 days using companies twitter $Symbol (Ticket with an $)

Hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.