Earlier this week Twitter launched a new framework for developers called Storm. The details of Storm are highly technical and I don’t pretend to understand exactly how it works (we’ll leave it to our awesome developers & co-founders Adrian @apetresc and Abhi @abh1nv for that). I do know this system is important to the future of mobile and web apps - especially related to social media, and here’s why.
Warning: the following is my attempt to describe Storm at a high level. If you just want to see the Emmys in action please skip to the video.
Storm allows you to analyze a stream of information in real-time. Hmmm… still too technical? Okay. First, lets talk about a “stream”. There are two ways to get information. One method is to simply ask for it, over and over again, which is a lot of work. You must continue asking to see if there is anything new available. The second method is one where new information is given to you, immediately, if and when it is available. This is obviously superior to asking repeatedly. Storm essentially helps computers handle the second case, where information is fed to the computer as a constant stream.
Receiving and storing information is easy, analyzing it on-the-fly is hard. Storm provides a highly scalable framework to help us analyze data from the stream as it arrives. We expected Storm to be powerful and quickly discovered it’s easy to use as well. The next question for us was how to experiment with it. And as a company of social media data geeks, this was the fun part.
The 63rd Primetime Emmy Awards were broadcast live the night before Storm was released. Adrian setup a process to capture tweets about the Emmys, ~188,000 tweets in total. We probably missed a few tweets, but that’s okay as this was not the focus of our experiment. TwitSprout built an internal tool to replay the stream of tweets as if the Emmys were occurring right now. We can also speed up and slow down the playback to simulate tweet volume from many different sizes of events.
Now let us suggest an interesting question and apply Storm to our massive pool of Emmy tweets to find the answer. We were curious which TV shows received the most buzz on Twitter. Determining this after the fact is very straight forward - simply add up the tweets and you’re done. The trick is visualizing the relevant tweets for each show as they occur on Twitter in real-time. Replaying the Emmy tweets through Storm actually highlights the crowds reaction on Twitter to key events happening on stage. You will notice a surge in mentions about Modern Family or The Daily Show when they win awards. Tweet activity is also influenced by what the speaker mentions on camera.
What you’re about to see is very cool. We spent an entire night working on this experiment. The first time we played the 188,000 Emmy tweets through our system the entire team was cheering for their favorite TV show as they competed for top rank.
To achieve these results we asked Storm to filter tweets by specific keywords and group these keywords into categories. Each category is a TV show. Below is a list of the TV shows and keywords used. We realize this list could be more refined & complete but it was very late and we were eager to finally test our experiment.TV Show Keyword Filters
GLEE (“glee”, “msleamichele”, “colfer”),
GAME_OF_THRONES (“#got”, “thrones”),
MODERN_FAMILY (“modern family”, “modernfamily”, “burrell”, “bowen”),
MILDRED_PIERCE (“mildred”, “winslet”, “pierce”),
BIG_BANG_THEORY (“big bang theory”, “big bang”, “sheldon”, “parsons”),
OFFICE (“office”, “carell”, “dunder”, “mifflin”),
SATURDAY_NIGHT_LIVE (“snl”, “saturday night live”),
PARKS_AND_REC (“parks”, “poehler”, “recreation”),
DAILY_SHOW (“daily show”, “stewart”),
I applaud Adrian’s execution on this. He learned Storm’s framework in less than 24 hours and programmed a very engaging demonstration. Adrian also recorded his code development as a live screencast to help other programmers learn Storm. Abhi is responsible for the awesome graph visualization, without which we would only see streaming lines of code :| Thank you!
I should mention that we have increased the speed of these tweets 60 times, meaning one minute in real time is equal to 1 second in the video (1min = 1 second). This allows us to replay a the Emmys in 2 minutes :)
Let’s get to it! Full screen the video to watch the results.
We would love to see your response in the comments below.
What should we analyze next? I’m excited to find out.
Dan, Adrian & Abhi