Programitically download Twitter data into GraphML using NodeXL

Mar 14, 2013 at 4:28 AM
Edited Mar 14, 2013 at 4:29 AM
Hi,

Is there any library to programatically download the twitter data into a GraphML provided by NodeXL library. If not, it will be great if anyone knows of such api which can do so.

Thanks
Coordinator
Mar 14, 2013 at 8:00 PM
We don't package that as a library, but the source code is available on NodeXL's Source Code tab at http://nodexl.codeplex.com/. The relevant code is in a project called "GraphDataProviders."

What exactly are you interested in? I don't know what you mean by "the Twitter data." NodeXL can import three types of Twitter networks: Search, User, and List. Are you interested in duplicating this functionality in your own application? If so, do you want your program to open up one of NodeXL's dialog boxes where a user can specify exactly what to include in the network (for example, what to search for in the Search network), or will your program just hard-code what should be included?

-- Tony
Mar 15, 2013 at 2:33 AM
Hi Tony,
I am trying to learn NodeXL in order to create an app which is not having any dependency on Excel. I have a winform project in which I am trying to use the NodeXL libraries to draw a graph out of the fetched twitter results. Initially I have started with the Search Twitter API, and want that all the users related to top 50 discussions for any particular keyword search gets visible in form of Graph.

I want to reuse the features built in the NodeXL libraries and try to contribute back to the community with an app. My future plans are to integrate this feature in my own website NetMe. This is a non commercial website which does Twitter based analysis. Currently I have implemented Gephi for analyzing Facebook pages at http://netme.cloudapp.net/DataVisualization.aspx

I feel it will be great if there is a library created for Social media interaction, that will give more robustness to the existing project and encourage more developers to contribute to this forum.

Nabarun
Coordinator
Mar 15, 2013 at 6:52 PM
I think you can reuse NodeXL's Twitter library without too much difficulty. I'll include instructions for doing so in my next post. Please note the following:
  • I haven't actually tried doing this, but I think it should work.
  • The Twitter importers will be overhauled in an upcoming NodeXL release, which might be sometime in the next two weeks. Twitter is discontinuing the version of the Twitter API that NodeXL is using (see https://dev.twitter.com/blog/planning-for-api-v1-retirement ), and we need to adjust NodeXL accordingly. The main changes are that you will be required to authorize NodeXL to use your Twitter account, which is optional today, and that getting follows relationships will be extremely slow due to stricter rate limits that Twitter is imposing.
  • The changes will be internal to NodeXL's Twitter library. The public calls for getting Twitter networks from NodeXL will not change.
-- Tony
Coordinator
Mar 15, 2013 at 7:01 PM
Edited Nov 20, 2013 at 4:37 PM
To reuse NodeXL's Twitter network importers in another application:
  1. Download the NodeXL source code.
  2. Open NodeXL.sln in Visual Studio 2008 or 2010. If you are using Visual Studio 2010, let Visual Studio convert the solution.
  3. In Solution Explorer, delete all projects except GraphDataProviders, ExcelTemplatePlugins, and Util.
  4. Modify TwitterAccessToken.GetAccessTokenFilePath() to fit your application's needs. NodeXL stores Twitter's access token in a file so the user doesn't have to reauthorize with Twitter every time she imports a network.
  5. Build the solution. The assembly you need will be at NodeXL\GraphDataProviders\bin\Debug\Smrf.NodeXL.GraphDataProviders.dll. Add a reference to this assembly to your own application.
  6. In your application, create a TwitterSearchNetworkGraphDataProvider object, then call its TryGetGraphDataAsTemporaryFile(out String pathToTemporaryFile) method. That method opens a dialog for specifying what to include in the network, gets the network from Twitter, and stores the results in a GraphML file.
  7. IMPORTANT STEP: Open the file "\Common\SocialNetwork\Twitter\Authorization\oAuthTwitter.cs" and replace _consumerKey and _consumerSecret with your own Twitter OAuth consumer key and consumer secret. The key and secret identify your application to Twitter. For instructions on how to obtain a key and secret for your application, see "how to get my api key" at https://dev.twitter.com/discussions/631 .
-- Tony
Mar 16, 2013 at 2:58 AM
Thanks a ton Tony....

This is indeed helpful. Let me explore this more and then provide my feedback
Nov 19, 2013 at 3:16 PM
Hi Tony,
I tried the instructions above and was able to build and use the GraphDataProviders library. However, TryGetGraphDataAsTemporaryFile is no longer a method of TwitterSearchNetworkGraphDataProvider but that of GraphDataProviderBase. Could you update the instructions?

Thanks much!
Coordinator
Nov 19, 2013 at 9:15 PM
I'm glad you got it to work.

TryGetGraphDataAsTemporaryFile() is indeed implemented in the base class, GraphDataProviderBase, but it is called on the derived class, TwitterSearchNetworkGraphDataProvider. I don't understand what needs to be updated--the instructions still look accurate to me. Please explain.

Thanks,
Tony
Nov 20, 2013 at 1:51 AM
Ok, I got that part to work, it was a syntax error that had prevented it from working. I have run into a different issue - I see the error "Problem occurred while attempting to authorize NodeXL to use your Twitter account. Details: The remote server returned an error: (401) Unauthorized. I had updated GetAccessTokenFilePath before building the NodeXL solution. I will look through related forum posts and try to work through this error.
Nov 20, 2013 at 3:49 AM
When trying the other option about using Twitter - I have a Twitter account and I have authorized NodeXL to use my account to import Twitter networks - I see the following error.
The network couldn't be obtained. Details: The Twitter web service refused to provide the required information. The stated reason was "unauthorized"
Any ideas how to get past this?

Thanks
Coordinator
Nov 20, 2013 at 4:21 PM
Edited Nov 20, 2013 at 4:37 PM
Sorry, I left out an important step in my previous instructions.

The source code contains a "consumer key" and "consumer secret" that tell Twitter which application is making requests--NodeXL in our case. The key and secret in the source code won't work for you; you'll have to request a new key and secret for your own application from Twitter.

Here is the additional step I just added to my earlier post:

IMPORTANT STEP: Open the file "\Common\SocialNetwork\Twitter\Authorization\oAuthTwitter.cs" and replace _consumerKey and _consumerSecret with your own Twitter OAuth consumer key and consumer secret. The key and secret identify your application to Twitter. For instructions on how to obtain a key and secret for your application, see "how to get my api key" at https://dev.twitter.com/discussions/631 .

-- Tony
Nov 20, 2013 at 5:31 PM
Thanks, I went ahead and added that. I am running into a different issue "The type or namespace name 'Util' does not exist in the namespace 'Smrf.NodeXL' (are you missing an assembly reference?)"
I have already ensured that the Target Framework for all the projects is set to the same .NET version. Is there anything that needs to be added besides GraphDataProviders, ExcelTemplatePlugins, and Util?
Coordinator
Nov 20, 2013 at 5:48 PM
Edited Nov 20, 2013 at 5:52 PM
It sounds like you have a "using Smrf.NodeXL.Util;" statement in your application's source code. If so, you should remove that statement, because there is no such namespace.

-- Tony
Nov 26, 2013 at 10:28 PM
Thanks Tony, I got it working.

-Vani