Twitter Import option not working properly

Sep 5, 2016 at 4:11 PM
Hello, and good morning.

I'm using NodeXL Pro to import content from Twitter (using search network option). I'm trying to import results for a very popular hashtag in Brazil, that certainly has had more than 18 thousand tweets last week (the hashtag in question is "#justiça", the name of a tv show and also the portuguese word for justice). It has reached trending topics last friday.

When I hit import though, only 787 tweets were imported. I know for a fact this quantity does not correspond to the reality. What I expected to happen was for NodeXL Pro to import the last 18 thousand tweets available on Twitter API (that lasts for at least 7 days, if I'm not wrong, or 30 if using firehose). I have used NodeXL in other ocasions for that same purpose and suceeded.

Could anyone please explain what is going on and if there is any way to fix it?

Thank you!

Sep 5, 2016 at 6:06 PM

Thank you for the interest in NodeXL!

Twitter limits all data access through its API.

Twitter will not deliver more than 18,000 tweets via the REST API in any query.

Twitter will not deliver more than 7 or 8 days of data.

NodeXL Pro provides access to the full Twitter API and is limited to 18,000 tweets or 7-8 days - which ever comes first.

When you perform the same query multiple times, you may get very different results. Just as you can never enter the same river twice, Twitter may have varying levels of activity or server load that leads them to deliver different sets and volumes of tweets.

Only a commercial relationship with one of the Twitter data resellers can get closer to a "complete" data set.

Note, different platforms have different approaches to data access limits. For example, Facebook provides access to data based on a rate limit but does not impose a historical limit - you can collect Page and Group data back to the dawn of time (or 2007).

You can enter advanced "date range" operators in the search term box in the Import from Twitter Search Network dialog box. For example, if you enter this into the search term box:

NodeXL since:2016-09-03 until:2016-09-04

...then you'll get the tweets between those two days. You can read about these and other operators by clicking the "Advanced search help" link in the dialog box.

However, Twitter severely limits how far back you can go. Here is what Twitter says about that:
"The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets."
(The "Search API" is what NodeXL uses to get tweets from Twitter.) The unfortunate result is that you will not be able to get the June-through-August tweets that you want. There is no way to do that in NodeXL.


Marked as answer by MarcSmith on 10/16/2016 at 12:00 PM
Oct 1, 2016 at 2:36 AM
Hello, Marc, and thank you so much for your answer.

Yes, I'm aware of those API limits. But the results I'm getting did not go back longer than 1 day and are as few as 800 tweets.

And the hashtag I searched for was a Trending Topic on the last 7 days, so I should've gotten the whole 18000 results, but that's not happening. It's like the software is not going any further than the actual day.

That's why I'm confused.

Thank you.
Oct 16, 2016 at 8:01 PM

I am checking back to see if you were able to collect a larger slice of data from the Twitter search API?

In some cases Twitter delivers more data in the days AFTER a topic is focus of peak activity.


Oct 31, 2016 at 2:55 AM
Edited Oct 31, 2016 at 3:15 AM
I am running into a similar issue. I'm trying to pull tweets but it is only coming back with less than 24 hours of historical data as well as less than 18,000 tweets. I also notice the timestamps seem incorrect. Pulled data shows tweets that occur at the same time, but when you view the tweets themselves they are posted as occurring at different times.