Final Reflection

Initially my project was meant to examine governmental control over the internet by looking at the January revolution in Egypt. On Twitter major events are generally tagged with what is called a hash tag. This means a keyword is preceded by the hash character (#). This groups tweets together by theme and allows for searches regarding that event or meme. For the revolution in egypt the most common hash tag was #jan25. My plan, then, was to gather tweets using that as my keyword.

I quickly ran into two problems. First, Twitter only gives access to a limited history of tweets. This meant that by the time I started my data collection in April the tweets that were sent out live from the event were no longer accessible to me. Secondly many of the tweets were in Arabic, which I do not read and which would not be easy for me to work with programmatically. So, my thought was to do this as a proof of concept and gather current tweets in English which were still dealing with the ongoing changes in Egypt.

My first plan to go about gathering tweets and creating a visualization was to use the Python programming language. I had had it recommended to me in the past as a good learning language and had previously begun learning it. However, when I met with Don Craig from the DXARTS program he recommended Processing as an easier, quicker way to go about this. Since it was another language I had heard was relatively simple to learn I looked into it. I quickly changed my mind about Python and switched to Processing as my language.

However, when I first looked into accessing the Twitter API it appeared as though I could just use a web browser to gather data. I spent one lab session doing so, but it was awkward and time consuming. So, I went online and searched for ways to gather tweets directly from Processing. I found a program by a woman who goes by Robotgrrl online called Simple Processing Twitter. It is an implementation of the Twitter4J Java library for Processing and is quite simple and powerful. I was able to use her code to write a program that gathered tweets from Twitter as they were being published.

Later in the term the perfect event presented itself on which to try my program. It was announced on May 2nd that Osama bin Laden had been killed in Pakistan. This was a major event all over the world so Twitter was very active. I modified my program to gather 100 tweets with the keywords “bin laden” every half second. I ran it until Twitter timed me out and was able to gather roughly 73,000 tweets. Although in the scheme of things this is a relatively small sample, it is more than enough for the proof of concept I’m working on now.

After gathering the data I needed to write software to sort it and generate 3D coordinates. At first I couldn’t figure out the best way to do this. I started chatting with a programmer friend of mine, Greg Williams, and he suggested making each tweet into an object. This suggestion was what allowed me to do what I wanted to do. I created an object that stored the username, date, and content of the tweet separately, then also had space to store the coordinates. This allowed me to sort the tweets, analyze the keywords, and store the coordinates, all in one place.

As written now my program sorts the tweets alphabetically by username to generate the x coordinate, then by time to generate the y coordinate. The z coordinate is created by analyzing the tweets for common keywords from throughout the sample. I used the top 15 keywords and assigned them numerical values. Then, each time a keyword appears in the tweet, a variable called weight is increased by the associated value. Weight then becomes the z coordinate.

The downside to this method of creating the z coordinate is that many tweets do not contain any of the top 15 keywords. This means that out of my 100 test tweets almost all are assigned 0 as the z coordinate. I expect this to change somewhat when all 73,000 are included, but I may need to either change how the coordinate is assigned or increase the number of keywords used. Somewhere between 50 and 100 may make the visualization more interesting. When getting into that large of a list, though, I’ll probably need to create a function to load them from a file rather than hard code them into the software. This will also make the program more flexible when doing other events or memes.

The major hurdle I faced in doing this project was learning to program in Processing. It has been six or seven years since I’ve done any programming (beyond making other people’s javascript work in my webpages), so refreshing myself on the concepts took a bit. I had also never done any object oriented programming, so learning about objects and how to write them was a fun challenge. Like with any new skill the more I did the better I got. I feel now that I have a solid foundation on which to improve my skills and I fully intend to do so in the future. As Processing is based on the Java programming language, moving to the more robust and powerful Java should be relatively easy. I’m excited for this new challenge.

Currently the visualization I’ve created only has 100 points. Also, because it is just points I’m not sure I can export it as a DXF file as I had planned. It looks from the Processing documentation that I need a triangle-based image to do that. I plan to experiment anyway because I need some method to translate these points into a file format that can be read by 3D imaging software. This is the immediate next step in my project.

After figuring out how to export the file, I’ll move it into Blender, the open source 3D imaging software I have. I should be able to create a solid piece based on the points. Once that is complete I’ll be able to send it out for printing and get back a physical artifact that represents this event as seen on Twitter.

Ultimately I’d like to do this for multiple events and even memes on Twitter. I’d also like to start experimenting with ways to color code the pieces so they are not just plain plastic.

I’d also like to rewrite the software so it groups tweets by network rather than alphabetically by username. So, if one user retweets another, their tweets will be near each other. I think this will result in much more interesting points than simply sorting alphabetically. I thank my friend Dave Proctor for this suggestion.

JavaTM 2 Platform, Standard Edition, v 1.4.2
API Specification
http://download.oracle.com/javase/1.4.2/docs/api/overview-summary.html

“Processing Forum”, accessed May 25th, 2011, http://forum.processing.org
This forum is open to anyone using Processing and is a great resource for finding answers to programming problems not covered in the documentation. If it hasn’t been done already, one can post one’s problem and get input from programmers from around the world.

“Java Language Specification 2nd Edition”, accessed May 25th, 2011, http://java.sun.com/docs/books/jls/second_edition/html/jTOC.doc.html
This is a resource covering the nuts and bolts of programming in Java. Very useful for advanced users of Processing wanting to go beyond the documentation on processing.org.

“Object Oriented Programming Tutorial”, accessed May 25th, 2011, http://processing.org/learning/objects/
This outlined the basics of what an object is, how it can be used, and how to create one. It was essential to writing my own object.

“Java Notes”, accessed May 25th, 2011, http://leepoint.net/notes-java/index.html
This is a tutorial for the Java programming language. Useful for basic to advanced concepts.

“Java 2s”, accessed May 25th, 2011, http://www.java2s.com/
This is a tutorial website for many programming languages. It includes a great Java tutorial as well as the documentation for the language.

“Twitter4j”, accessed May 25th, 2011, http://twitter4j.org/en/index.html
Twitter4j is the Java library that allows access to the Twitter API. It is what Robotgrrl’s Simple Processing Twitter is based on.

“Simple Processing Twitter”, accessed May 25th, 2011, http://robotgrrl.com/blog/2011/02/21/simple-processing-twitter/
This code allows direct access to the Twitter API from within a Processing program.

Shiffman, Daniel. 2008. Learning Processing: a beginner’s guide to programming images, animation, and interaction. Amsterdam: Morgan Kaufmann/Elsevier.
This is a step by step guide to learning Processing. It starts with very basic concepts like drawing shapes and lines, then moves on to more advanced aspects of the language.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *