Twitter API whitelisting

Jun 2, 2010 at 8:16 PM

Hi, I need Twitter to whitelist my use of Twitter API but they're asking "specify which API methods you'll be calling, and at what frequency".

Can anybody help me out here with the methods part? I'd like to do mostly research which means ad-hoc usage, but also develop Excel mashups with Klout.com API and such.

Thanks, NodeXl is awesome! :-)

Jiri

Jun 3, 2010 at 1:22 AM
Edited Jun 3, 2010 at 1:22 AM

Jiri:

NodeXL uses the following Twitter API methods:

* statuses/friends
* statuses/followers
* users/show
* search

The number of times these methods get called varies greatly with the NodeXL Twitter dialog box you use, the options you specify in the dialog box, and of course how often you use those dialog boxes, but here are two typical examples:

* Twitter User's Network, add a vertex for each person followed by the user, include 1.5 levels, limit to 100 people: 101 calls to statuses/friends, 1 call to users/show.

* Twitter Search Network, search for people whose tweets contain some popular term, add an edge for each follows relationship and mentions relationship, limit to 100 people: 2 calls to search, 101 calls to statuses/friends.

So you can figure about 100 API calls each time you use one of NodeXL's Twitter dialog boxes with some typical settings.  Multiply that by the number of times per day you think you might use the Twitter dialog boxes, and that will give you the number of API calls per day you'll be making, which is the "frequency" that Twitter is asking you for.

Please let us know if you succeed in getting whitelisted.  From what I've heard, not a lot of people have.

-- Tony

 

Jun 4, 2010 at 11:08 PM

Tony, thanks a bunch, I have succeeded!!! :-)

Jun 7, 2010 at 7:25 PM
Hi Tony, I also requested a Twitter whitelisting and got "We need more information to process your request. Please re-apply and specify which API methods you'll be calling, and at what frequency.". I am not completely clear on the explanation above in terms of my needs - I am currently using the 'Import from Twitter Search Network' function in NodeXL, mainly for just '"Mentions" relationship in tweets' - for ad-hoc usage / research. The max limit it allows me at the moment (without whitelisting) is 500 people (using the 1000 people setting doesnt seem to deliver any more). If I wanted to find more people mentioning the search term / have coverage of more tweets - how would the whitelisting help (if I can get it) - would I need to make repeated calls and then filter out duplicates in Excel / or could I make a bigger search in one go? Just to clarify - in the example I just gave, how many API calls would that count as? Many thanks in advance - apologies if unclear - to quote jiribrazda, "NodeXL is awesome"!
Jun 7, 2010 at 10:56 PM
Edited Jun 7, 2010 at 10:56 PM

see2monkey:

The options you are specifying are much less API-intensive than the options I used in my June 2 post, and you're probably not running into Twitter's rate limits at all.  You'll know if you are, because the network download will be interrupted with a message that says something about "a likely cause is that you have made too many requests in the last hour."  If you are not getting this message, then you don't need to apply for Twitter whitelisting.  (If you are indeed getting this message, let me know and I'll provide you with more API details that you can give to Twitter.)

The apparent limit of 500 people you're seeing is probably due to the way NodeXL's Twitter Search Network operates.  NodeXL asks Twitter for 100 tweets at a time, then filters out duplicate people.  It continues this process until N unique people have been found, where N is your "limit to N people" setting; or until 1,500 tweets have been received from Twitter, whichever comes first.  Even if you set N to 1,000, there may be only 500 (or 450 or 300...) unique people in the 1,500 tweets Twitter provides.  This behavior has nothing to do with Twitter rate limiting, although rate limiting could kick in if you ask for enough search networks in a short period of time.

-- Tony

Jun 7, 2010 at 11:11 PM
Tony - many thanks for such a swift response - very helpful - the only additional question I have is that in the download - the twitter user details (e.g. total tweets, number of followers etc) is blank in significant number of the vertices - is this a common problem with the API or something in my method - this is a shame since the total # of followers per user is quite important for the analysis I am doing. Many thanks again for your support and a great application. Daniel
Jun 8, 2010 at 12:25 AM

Daniel:

You're checking the "Add statistics columns to the Vertices worksheet" option, correct?  That's the only way to get the statistics you mentioned.

Please send me an exact search term ("search for people whose tweets contain") that yields blank statistics and I'll investigate.

-- Tony

Jun 8, 2010 at 7:59 PM

Hi Tony,

 

Many thanks for your response. Yes - I am including "Add statistics columns to the Vertices worksheet" option. I encountered the Twitter rate limit yesterday after your last reply, but have tried several more imports today (always 'Import from Twitter SEARCH network...') , with varying options and still only getting at best statistics for some, but not all nodes, and my last 3 attempts have returned no statistics at all for any nodes (a shame - as everything else seems to be working fine - and finally got my head around this great tool).  As a test, I also ran a an 'Import from Twiter USER's network' - and that does return statistics for all nodes - but unfortunately I need the Twitter SEARCH import (this also suggests to me it wasnt to do with rate limits - as the USER's network option returns much more data).

 

My 'Import from Twitter Search network...' criteria are:

 

1. Search for: "mflow" [but tried with other search terms, including the default 'nodeXL' and same problem]

 

2. Add an edge for each: Checking Mentions relationship [NOT checking requesting follows / replies at the moment]

 

2. Limit to 500 people [tried with 100 people and ended up with no stats for any nodes
columns]

 

3. Add a tweet & Add statistics both checked

 

 

Having encountered the Twitter rate limit now - I was interested in confirming what usage I should specify (based on perhaps 10 of the above type queries a day) - and also whether in your opinion I would need to provide a static IP (I dont have - so hoping not). Many thanks in advance, in great anticipation. Daniel

Jun 9, 2010 at 12:39 AM

Daniel:

Twitter omits some statistics when you don't provide whitelisted credentials.  When I used my whitelisted credentials with the NodeXL settings you listed above, I obtained statistics for everyone.  When I omitted my credentials, many statistics were missing.  I don't know why Twitter does this.  It's especially odd, because I can go to the Twitter.com Web page for an individual and get the same statistics that are denied to me when I go through NodeXL.

So the solution is for you to apply for whitelisting after all.  Tell them that you are using the following APIs:

* statuses/friends
* statuses/followers
* users/show
* search

...and that you are making about 100x10 = 1000 API calls a day.  (Each NodeXL search network takes roughly 100 calls when you check "Add statistic columns to the Vertices worksheet.")

-- Tony

Jun 9, 2010 at 12:39 AM
Edited Jun 9, 2010 at 12:40 AM

Oh, and I haven't heard of anyone needing a static IP address.

-- Tony

Jun 9, 2010 at 4:17 PM

Great - thank you again - will let you know if I am successful.

Daniel

 

Jun 11, 2010 at 2:36 PM

Twitter Whitelisting approved! Many many thanks Tony.

Daniel

 

Jun 11, 2010 at 3:59 PM

That's good news.  Thanks for letting us know, Daniel.

-- Tony

Sep 21, 2010 at 10:12 AM

Hi Tony,

I was using the Twitter import fine for a couple of months but since taking some time off from this over the last 1-1.5 months when I do a Twitter import (I sign in) and select to import statistics - I get all the edges / vertices but none of the Twitter statistics (I get the image file and tweet, but none of the followers information etc) - has something changed? Help please.

Many thanks as always,

 

Daniel

Sep 21, 2010 at 5:13 PM

Daniel:

I copied your post to http://nodexl.codeplex.com/Thread/View.aspx?ThreadId=228025 and answered it there.

-- Tony