twitter rate limit and vertex relationship??

May 6, 2013 at 11:00 PM
Hi, I am trying to configure some data from twitter on posts that mention, include, tweet or retweet posts with the keywords "constitution pbs". I have done a manual search on twitter and know that there are hundreds of such posts (some of them being mine). However, when I try to import data from twitter, I am almost immediately given the "rate limit exceeded" warning and have to wait 15 minutes.

After that, I only have partial information which gives me data to about 50 rows (my tweets do not show up on the data for some reason). I know there is tons of data but how can I import all of it? I have read various forums on this issue but haven't found a solution. After I click "import from twitter search" a window pops up with many options. I have tried various options but they all yield the same result--very little data.

Also, in the relationships column, I only see "following" concerning the first and second vertex. The first time I attempted this I got proper data which included "mentioned, tweeted" and etc. How can I get that data back?

Hope this wasn't too confusing and thanks for all the help!
Coordinator
May 7, 2013 at 2:06 AM
Hello!

It could be that you have selected the "Add an edge for each: Follows relationship (slower)."

That "slower" is an understatement.

Which version of the application are you running? Are you logged in when you query?

Regards,
Marc
Coordinator
May 7, 2013 at 2:54 AM
The map of the Twitter search network query for constitution pbs is here:
http://nodexlgraphgallery.org/Pages/Graph.aspx?graphID=4532
Image
Coordinator
May 7, 2013 at 3:28 AM
Edited May 7, 2013 at 3:31 AM
Kyoki:

You are using a recent version of NodeXL that severely limits some of the information you can get from Twitter in a reasonable period of time. This is due to changes imposed by Twitter on programs like NodeXL. A discussion about this issue can be found at http://nodexl.codeplex.com/discussions/437841.

The limit you are running into has to do with "following" relationships. In the Import from Twitter Search Network dialog box, uncheck the edge option for such relationships and you will no longer encounter this 15-minute rate limit. Of course, you won't get followed edges either, but there is nothing NodeXL can do about that.

If you want "mentions" edges, "replies to" edges, and edges that aren't mentions or replies-to (which NodeXL arbitrarily calls "Tweet" edges), then check the corresponding checkboxes for those edges as well. Those types of edges are not rate-limited.

Note that the graph Marc created for you was done with an earlier version of NodeXL that was not subject to the same severe rate limits as recent versions, and so he was able to get followed edges for you. If you try the same thing, it could take hours or even days to get the same network.

-- Tony
May 7, 2013 at 3:32 PM
Hi Marc and Tony,

Thank you so much for all your help and input! Tony you were right, I unchecked the edge option for relationships and I got all the data I wanted right away. Also the relationship column now tells me whether it was a tweet, mention, etc., and that's what I was looking for. Before, I was just getting "follows" and for the "relationship date" I was just getting that exact date and time when the data was imported--so all the data had the same date. Now I got all the correct info. Thanks again!

Marc, thanks a bunch for the graph! Looks great so far and am looking forward to really examining it!

Will let you know if I have any more questions.
Thanks to both for such quick replies.
-Aliya
May 22, 2013 at 6:09 PM
Hello! I have one more question. I was reading on another post that you cannot choose specific dates for twitter data. However, on a few NodeXL projects I was working on a few weeks ago, I got twitter results based on a span of about 3-4 days. (Like if I did NodeXL on Thursday, twitter would give me data from Monday, Tuesday, Wednesday, and Thursday). But when I attempted this again, I am only getting twitter data from today (5/22/2013). Can you please explain why I'm not getting data from yesterday and the day before? Thanks!

-Aliya
Coordinator
May 23, 2013 at 1:48 AM
Aliya:

Please see if this discussion answers your question:

https://nodexl.codeplex.com/discussions/442022

-- Tony
May 23, 2013 at 3:38 PM
Tony,

I did try the "since-until" method but it yielded the same results as yesterday--I only acquired data for 5/22/2013. It gave me about 101 rows of data when I should have gotten more than a thousand rows. I went ahead and did the same thing but changed the dates from 5/19/2013 to 5/20/2013 and only got 14 rows of data, when I know I should have received hundreds. I never had this problem before so I'm not sure what's causing it? All the previous times gave me data on a span of about 5 days.

Any other help would be greatly appreciated, thanks!

-Aliya
Coordinator
May 23, 2013 at 4:31 PM
Twitter controls what data it provides, and it delivers data with several limitations. First is the limit on total tweets delivered, which is set at 1,500 tweets under Twitter API 1.0 and has risen to 18,000 tweets under Twitter API 1.1. If your topic was discussed slowly at first and then became popular you could query at time 1 and get 4 days and query again at time 2 and only get 1 hour, if 1,500 tweets occurred in that time. Second, the total collection of tweets is somewhat random, not all tweets are delivered the same for two identical queries. Third, repeating a query at a later time is likely to generate different results.

The latest versions of NodeXL now make use of the 1.1 API which has changed the rules again, making the query for the "Follows" relationship pretty much impossible to use. On the positive side, the new versions of NodeXL using the latest Twitter API can get larger volumes of tweets (when they exist).

Regards,
Marc