Twitter User Network Import issues

Jul 19, 2013 at 4:07 PM
Edited Jul 19, 2013 at 4:08 PM
I am having consistent difficulties with importing any network at all from the Twitter API.

Here's the settings for the import:

Both followed and following the user
Followed/following relationship checked
1.5 degree network, no limit

My issue is that NodeXL refuses to pause for the appropriate amount of time. Every time. The API dictates that NodeXL pause for 15 minutes before retrying it's request to avoid being told "no" by Twitter.

For literally any search query I do, for any user handle, it does not pause for 15 minutes. It will make its initial pull from the API and then pauses for between 10 and 14 minutes, generally around 12 minutes and 30 seconds. The most recent attempt only paused for 11 minutes and 25 seconds. It then tells me that only a partial network has been obtained, and that there was a problem getting the rest of the network:

"Partial Network

_Part of the network you requested was obtained, but there was a problem getting the rest of the network.

Do you want to import the partial network?_"

It gives an option for details, where I am provided with this message:

"NodeXL

Getting a network can involve many information requests to a Web service. in this case, 33 requests were made and 1 of them was unsuccessful.

(Note that unsuccessful requests might have led to additional requests if they had succeeded, so it is not possible to calculate the percentage of the complete network that was actually obtained.)

Here are the details for the most recent unsuccessful request:

The twitter Web service refused to provide the requested information. A likely cause is that you have made too many twitter requests in the last 15 minutes. (Twitter limits information requests to prevent its service from being attacked. Click the 'Why this might take a long time: Twitter rate limiting' link for more details.)
"

I am running on Win 7, 64 bit, with Excel 2010
The clock on the computer is auto-updated to windows live, it is correct, it is set for EST, etc.

I have tried previous versions of NodeXL to no avail.

Is there some corruption with my copy of NodeXL? I've tried re-downloading it a number of times, and I am just completely frustrated and baffled by it. Any help would be appreciated. Thanks!
Jul 19, 2013 at 6:30 PM
Edited Jul 20, 2013 at 5:09 PM
This is a known problem with getting large networks from Twitter. There is a long discussion about it in "Partial network problem in Twitter" at http://nodexl.codeplex.com/discussions/447703, but I can summarize it by saying that Twitter seems to arbitrarily reject further requests at random times, and that I don't know of a way to fix this in NodeXL. I would actually like to remove the NodeXL features that can lead to this condition, because they just aren't working reliably today.

Regarding the 15-minute pauses, NodeXL is required to wait 15 minutes after the first request in a 15-minute "window," not 15 minutes after the most recent request. So if the first request is at 1 PM and the limit is reached at 1:05 PM, then Twitter tells NodeXL to wait until 1:15 PM for the next request, not 1:20 PM.

And that's the way it works, by the way--Twitter tells NodeXL when to start again, as opposed to NodeXL starting a timer. For a while I thought that clock synchronization issues might be causing the problem, but then I noticed that Twitter sometimes kicks NodeXL out on the very first request in a network, and that doesn't involve clocks at all. That lends weight to my theory that Twitter protects its servers with a rate limiting scheme that is sometimes augmented by outright rejection, based on server loads.

-- Tony
Jul 19, 2013 at 6:49 PM
Okay, thanks Tony!

I appreciate the response!

Yeah, I saw that forum there, and didn't know if I was missing something.

I'll keep at it and see what happens. I'll let you know!

Any idea if contacting Twitter would produce any kind of solution?
Jul 20, 2013 at 5:07 PM
I did try that:

https://dev.twitter.com/discussions/18999

Unfortunately, I didn't get any responses, which really surprised me. The problem I described seemed like an obvious design issue to me, and I expected someone on Twitter's staff to straighten me out with a simple explanation. Comments from other application programmers could have been useful, too, but perhaps not many programs are attempting to get large networks from Twitter in NodeXL's manner, and other programmers haven't run into this.

-- Tony