Twitter Refusal

Nov 30, 2010 at 10:53 PM

Sorry to have another question so soon, but I've received an error message from NodeXL:

.....................

The Network couldn't be obtained:

The Twitter Web service refused to provide the requested information. A likely cause is that you have made too many requests within the last hour...

......................

I take it that I got the message because I am not whitelisted.  Is there a way to lower NodeXL's request number per hour so that I don't get kicked-off? I'm running the latest version, 1.01.156

 

Patrick Glaser

Director of Research Standards

Marketing Research Association

110 National Drive, 2nd Floor

Glastonbury, CT 06033-1212

Ph:203.300.5563

Fx: 888.512.1050

patrick.glaser@mra-net.org

 

 

Nov 30, 2010 at 11:09 PM
Edited Nov 30, 2010 at 11:20 PM

Patrick:

I believe that Twitter is temporarily misbehaving, and that if you try again later today or tomorrow the problem may resolve itself.

If you're interested, this is how NodeXL and Twitter normally work together:

1. When NodeXL reaches Twitter's limit, Twitter tells NodeXL to pause for a certain period and then try again.

2. NodeXL pauses for that period.  (It tells you "Reached Twitter rate limits.  Pausing until ..." during the pause.)

3. NodeXL wakes up and continues getting information from Twitter.

What I think is happening is that when NodeXL wakes up, Twitter still refuses to provide additional information, and NodeXL gives up.  Twitter may be doing that because its servers are overloaded.

If this still occurs tomorrow, please send me the exact settings you are using in the Import from Twitter Search Network or User Network dialog box, and I'll try getting the network from here.

-- Tony

Dec 1, 2010 at 12:40 AM

Interesting, have you heard of this type of thing happening before?  This occured to me about a month ago as I was first experimenting with the NodeXL Twitter import function.  I tried again today for the first time since it had happened, and it re-occured.  I'll wait and try again in a day or so, but suspect that it may be a more permament issue that I'm experiencing with Twitter.

I'll let you know.  Thanks for all of your help.

Dec 1, 2010 at 12:49 AM

Not in this exact manner, no.  However, Twitter reserves the right to limit requests whenever it needs to, so that's why it is my first guess.

But it could be something else, too, which is why you should send me the exact values of all your dialog box settings tomorrow if it still occurs, and I'll try to reproduce the problem and see what's going on.  In the meantime, I'll try getting some random large Twitter network with a non-whitelisted account and see what happens.

-- Tony

Dec 1, 2010 at 9:57 PM

Try searching the popular Twitter user's network "neiltyson." I've had the request fail while trying to pull much smaller accounts, but most recently had it fail with "neiltyson" while using the following specs:

-Add vertex for both followed/following

-Add an edge for followed/following (but not "replies to" or "mentions")

-1.5 levels

-Add a latest tweet colum

-Linked Twitter Account

One thing I noticed is that I repeatedly get messages stating that Twitter has refused the request, then NodeXL pauses for 1 second and re-tries.  It eventually pauses again for a longer period once the non-whitelisted limit is reached.

 

 

Dec 1, 2010 at 11:05 PM

Patrick:

I'll try it here with the information you've provided.

The occasional one-second pauses are to be expected.  They just indicate a glitch in the network or in the Twitter servers, and NodeXL handles that by briefly pausing and then retrying the request.

You didn't say whether you are still getting a "The Network couldn't be obtained" message after the "Pausing until..." pause.  I assume that the problem persists.  If so, is the clock on your computer set accurately?  Twitter tells NodeXL when to wake up and continue requesting information.  If Twitter's clock and your computer's clock are significantly out of sync, NodeXL can wake up too early and be refused by Twitter again, at which point NodeXL quits.

-- Tony

Dec 2, 2010 at 2:30 AM

The clock on my computer is set correctly.  But, the Twitter account that I'm linking through is set to Pacific time, whereas I'm located in the east.  I will adjust and let you know if this solves the issue. 

Dec 2, 2010 at 2:56 AM

That should not be necessary and could make things worse.  Twitter tells NodeXL when to wake up in UTC time, which is independent of anyone's location.  If your computer's clock and time zone are set correctly, then everything should just work.  (I forgot to tell you to verify your computer's time zone as well.)

I'm currently running your query, which is a long one.  It has successfully paused for about an hour and resumed twice so far.

-- Tony

Dec 2, 2010 at 5:21 PM

Yes, I ran the same query last night and saw it pause and resume once before leaving the room and checking back this morning.  It had kicked me off at some point.  Let me know how things turned out on your end.  Assuming your query goes well, perhaps I should try to make a new Twitter account or try from a different computer, etc.?

Dec 3, 2010 at 6:29 PM

It's a huge query, and I had to cancel it after twelve hours because I needed the computer for something else.  But I never got the "The Twitter Web service refused to provide the requested information" message.

I can think of only three reasons why you would get this message when I don't:

1. Your clock is set incorrectly.

2. Your time zone is set incorrectly.

3. Twitter is treating your account or IP address differently from mine.

To test whether 3 is the case, you can try NOT using your Twitter account (check "I don't have a Twitter account.").  The query will take longer, because the rate limit will be 150 requests/hour instead of 350 requests/hour, but if it runs successfully overnight then it tells us there is some account-dependent behavior on Twitter's end.

-- Tony

Dec 3, 2010 at 8:27 PM

Thanks for all of your help, Tony.

I've restarted the query and it has been running fine, without failing, for about 24hrs. (It is huge.) I'm not sure why it seems to be working now and failed previously, but I'll keep your tips in mind if I encounter problems in the future.  Thanks again and all the best. 

I've work for a trade association (opinion research) and have written about NodeXL to our members (2000+), so hopefully some of them will begin to take advantage of your software.

Jan 30, 2011 at 3:36 PM

Tony,

I won't impose on you anymore, but figured your could use the info. The problem has recoccured consistently over the past 2 months- greatly hindering the functionality of the Twitter NodeXL function. I've confirmed that my own clock and the Twitter time zone's are correct. I've also attempted to collect data using a non-Twitter linked account (# 3 above). Not sure what the issue is, but maybe worth considering a depper investigation if others have experienced the same issue. 

Jan 20, 2012 at 1:09 AM

Tony and Patrick,

I am a NodeXL rookie but yesterday,i faced exactly the same problem as Patrick . I tried to import data from a huge Twitter user`s network (''CathyPap'')several times and i received the same error message with Patrick every time i tried. I used the following specs :

-Add vertex for both followed/following

-Add an edge for followed/following (but not "replies to" or "mentions")

-1.5 levels

-Add a latest tweet colum

-Linked Twitter Account

Since this discussion between you took place over a year ago, i would like to know, did u find any solution to your problem? Could anyone help me?

 

Thanx

Jan 20, 2012 at 1:20 AM

Patrick,

I am a NodeXL rookie but yesterday,i faced exactly the same problem as you did. I tried to import data from a huge Twitter user`s network (''CathyPap'')several times and i received the same error message with you every time i tried. I used the following specs :

-Add vertex for both followed/following

-Add an edge for followed/following (but not "replies to" or "mentions")

-1.5 levels

-Add a latest tweet colum

-Linked Twitter Account

Since this discussion between you  and Tony took place over a year ago, i would like to know,since then, did u find any solution to your problem? Could you provide any help ?

 

Thanx

 

Feb 16, 2012 at 7:22 AM

Has anyone found a solution to this persistent problem?  

Time is set properly, so that's not the problem. Used my twitter account as well as the NodeXL one, both produced the exact same problem.

I'm very keen to use NodeXL, but this problem is a show stopper for me.

Hope there is a solution somewhere, somehow. 

 

H. 

 

Apr 19, 2012 at 5:07 PM

The most recent NodeXL release (version 1.0.1.209, on 2012-04-17) includes this change:

* If Twitter refuses to provide more information even after NodeXL pauses for "rate limiting," you will now be given the option to import the partial network that was obtained at that point. Previously, the partial network was discarded and all your time was wasted.

You can get the most recent release at http://nodexl.codeplex.com/releases.

-- Tony