It's all a series of tubes, Part 2: And what is a Pipe if not a tube?

In Part 1 of this series, I talked about how I've started putting out a lot more information to different groups and how I'm using Ping.fm to make my outgoing flow easier to manage.  Now I needed to take care of everything coming in.  As mentioned previously, I'm using an RSS reader (Google Reader, to be specific) to manage news, and I wanted to try and leverage that technology.  If for no other reason than I look at my reader multiple times per day to keep up, it seemed to be my best option. 

 

Twitter seems to concur since they export everyone's tweets as an RSS feed.  I decided that making these feeds easier to manage would give me the opportunity to try out a technology I'd read about a few years back, but never really put any effort into: Yahoo Pipes.  Yahoo describes Pipes as "a powerful composition tool to aggregate, manipulate, and mashup content from around the web".  With no programming, you can use Pipes to aggregate or filter data from the web in a way that's meaningful to or works for you. 

 

Clicking the Create Pipe button brings us into the Pipe editor.  Like I said, there's no programming involved in the creation of a Pipe.  You simply drag operators or inputs from the left onto the canvas and you can visually see how your data will flow and be manipulated.  At the bottom is the debugger which will show you what your data looks like at any step in the process.   Since this first Pipe I'm creating is just to aggregate multiple Twitter feeds into one, I've dragged the Fetch Feed tool onto the editor page.  At the bottom, another operator appears called Pipe Output.  This represents your finished product.

 

 

Now that I've got my fetcher added, I need to populate it.  I head over to Twitter.com and visit the pages of those people I'm interested in following most closely.  On everyone's page, you'll find a link to an RSS feed.  Simply right-click it and choose "Copy link location".  Paste the first one into the URL field of the Fetch Feed bubble.  To add additional feeds, click the "+" above the URL box to add an additional field. 

 

 

If I click on the Fetch Feed source, it turns orange.  The really interesing bit happens in the debugger, though:

 The fetcher, once highlighted, does its job and you see the result.  It has fetched all of the items from the RSS feeds and is showing me what my output would be if I ended the chain here.  Unfortunately, at the moment, only one person has tweeted recently enough to show you.  But, were there tweets from other people, you'd see all of the intermingled into one continuous stream. 

Since this is just supposed to be a simple aggregator, I'm going to end it here.  To do so, simply click the little bubble on the bottom of the Fetch Feed source.  You'll see the bubble on the top of the Pipe Output tool turns orange.  This is to let you know that it's an acceptable place to terminate the pipe that's now coming out of the bottom of the feed fetcher.  Simply drag that pipe down and drop it onto the output tool and a link is formed.  If you then click on the Pipe Output tool, you'll see the debugger refresh and the same data as before will display.  That's it, we've now got a simple aggregator for our Twitter feeds.  Hit the Save button, give it a name and we're done!

 So, now what do we do with this?  Well, click on My Pipes at the top of the page.  Your new Pipe will show in the list.  Click it, and you should see the tweet feed.  All you need to do now is right click the "Get as RSS" link, choose "Copy link location" and paste it into your feed reader.  If you use Yahoo or Google Reader, it's even easier: just click the button.

Now, this was just a simple first Pipe to get our feet wet, but it really shows how powerful this tool can be.  It gets even more powerful when you start to take advantage of the filtering capabilities as well.  But, that's for Part 3...