Specifying Which Twitter Followers to Collect

Oct 19, 2012 at 9:49 PM

As I understand it, from reading through these discussions, when you download a Twitter User's network through NodeXL, it starts with the oldest followers (I believe Marc said it is based on when someone started following the user).

For people with thousands of followers, this means I only ever seem to get their oldest followers, as invariable after one or two or at most three "rounds" of downloading and hitting the rate limit, NodeXL inevitably fails to reconnect, leaving me with only a part of the network.

I really appreciate the recent update that allows me to at least save the partial network when that happens, but there doesn't seem to be anyway to restart the collection, starting from the last person successfully collected.

So, say I successfully get the first 1100 followers before Twitter grumpily kicks out my NodeXL session...  Is there anyway to ask Node to get the rest of that person's followers starting with follower 1101?  I can do this when use a script to to access the API myself, but as far as I can tell there is no way to ask NodeXL to "remember" the last follower collected.  Am I right about that?

And if so, how exactly are people collecting the huge follower networks I see in some of the Gallery files?  I don't think I've ever succeeded in having Node get more than about 500 followers before failing.

Any hints?




Oct 22, 2012 at 1:06 AM
Edited Oct 22, 2012 at 3:40 PM


You cannot ask NodeXL to get the rest of a person's followers starting with follower 1101, for example.

Most, if not all, of the huge networks you are seeing on the NodeXL Graph Gallery were collected by people who have been "whitelisted" by Twitter.  When you are whitelisted, you do not run into the severe rate limits that Twitter imposes on everyone else, so NodeXL doesn't have to pause and resume after an hour.

Unfortunately, Twitter will no longer add new people to its whitelist.  They stopped doing that in February 2011.

We tried to work around the rate limit by pausing, but I've heard from many people, including you, that Twitter sometimes refuses to continue, even after the pause.  Can you tell me the error message you are getting when NodeXL fails to reconnect?  You can often press Ctrl-C to copy the entire message, including the details, to the Windows Clipboard.

In general, I don't think NodeXL works well for getting large Twitter networks unless you are whitelisted.  The long pauses and the flaky "continue" behavior make the process painfully slow and unreliable.

-- Tony