February 7, 2007

Yahoo! Launches Pipes

Nik Cubrilovic

114 comments »

yahoo pipes

It takes effort to explain the significance of a new product when the immediate benefit to consumers may not be so obvious, and the awkwardly named “pipes” from Yahoo! is no exception. The product name is taken from the world of UNIX where a pipe is a conduit for the transfer of data between applications, while with the Yahoo product it is a conduit for data between web services. In a basic form Yahoo! Pipes allows you to take data from one or more sources and to bring it together, for example - to aggregate a group of feeds.

But Yahoo! Pipes goes beyond what just pipes are and what pipes do though as the application provides functions (or as they are called in the app - modules) that will perform a variety of different actions. There are modules available to prompt the user for input (a variety of input types), different operators to count, loop, cut, count, sort and merge data along with a variety of string and date functions. Because of this already broad base of available functions, Yahoo! Pipes is more akin to a shell scripting environment for the web rather than just a simple conduit between applications. It works like a visual procedural programming language with the output of the process dropping out at the bottom, in the form of text output, RSS, SMS alerts of even JSON. You can use feeds, user input or other pipes as input.

The beauty of the application is with its simplicity - a user can take any sources, user input requests or the above mentioned module and drag+drop them into place and then connect the pipes. Within minutes I had built an application (also known as a pipe, they should probably change the name as not everything can be a pipe) that would search for ‘Techcrunch’ in a variety of feeds, bring that data together, sort it and filter it for unique results. I saved the application and published it, from where I can now execute it at any time and receive the output in a variety of formats. I can take a copy of an existing pipe (application, argggh) and use it as a base template for my own pipe and I can browse an existing library of pipes.

Pipes can take any feed as input, and combined with the already available list of functions proves to be very powerful - my mind is still buzzing thinking about all that can be done with Pipes. I think some of the terminology needs to be cleared up, there needs to be a better introduction on the main page - but besides that this product is fantastic. It was inevitable that such a product would be released, and it is very good for Yahoo! that they managed to be the first of the big web companies to release such a product. The fact that they include Google Base as a default source in Pipes shows that the web is much more about interoperability than the desktop ever was or ever will be.

See Anil Dash, Tim O’Reilly and Jeremey Zawodny for more.

pipesscreen.png

  • Sphere It

Trackbacks/Pings (Trackback URL)

  1. web output
  2. 苦牢之最後一年
  3. Michael Kimsal's weblog
  4. The RageNET Weblog
  5. Joseph Scott's Blog
  6. @Laurent's
  7. nonsmokingarea.com
  8. Time Savers Part 1: Four Ways To Listen Faster | Groundhog Day
  9. Personal weblog of Robert Beekman » Blog Archive » Yahoo pipes
  10. Wesley Tanaka
  11. Ubik - » La rivoluzione di Yahoo Pipes
  12. Mark Beveridge » Yahoo! Pipes
  13. 阿土伯程式大觀園
  14. Ceyhun AKSAN - Görsel İletişim & Grafik Tasarım · Yahoo Pipes
  15. krizdabz BLOG
  16. Cohenville » Blog Archive » Build your own mashup with Yahoo Pipes
  17. Salim Ismail To Head Yahoo BrickHouse
  18. Maya on Interaction » Blog Archive » I’ll be your website
  19. Tumblr Custom RSS via Yahoo! Pipes « Kiyo’s Blog
  20. Microsoft Popfly: un createur de mashup Silverlight at (no)wire .beta
  21. PopFly properly launches Microsoft’s Silverlight with Style! « NI-Limits Blog
  22. Popfly: Sign of a Closer Microsoft-Yahoo Relationship? : Forecast-Blog
  23. JackBe Raises $9.5M, Provides Mashups for the Enterprise | Article Blog
  24. ‘Yahoo Pipes’: A brilliant vision but not a blockbuster! What’s the lesson! « Experience-Is-King
  25. juego instantaneo portal web
  26. The OPLIN 4cast » Blog Archive » OPLIN 4cast #42
  27. advance cash loan personal cash advance loan with savings account
  28. free cricket wireless ringtones
  29. ringtones for prepaid cell phone cell free phone prepaid ringtones verizon cell free phone prepaid ringtones sprint
  30. kev/null - The Pipes Team Shout Out

Comments

RSS feed for comments on this post.

  1. Hasan Jafri

    Pipes I likes, but I’m with Nik on the name not being so hot. It sounds medieval.

  2. Jeremy Zawodny

    Nik,

    Glad you like it. It’s definitely not a consumer product in the sense that something like Yahoo! Messenger is.

    I think the “immediate benefit to consumers” will not be immediate, since Pipes is an enabling technology platform for a whole new class of services.

  3. visible.mobi

    Finally a great product from Yahoo !

  4. Tom Churm

    I think the power of Yahoo Pipes will become apparent once you begin combining the Pipes with other widgets and services.

    For example, feed your Yahoo Pipes into my RSS2PDF service at http://rss2pdf.com to automatically generate a PDF version of it.

    Here’s an example using Nik’s TechCrunch example Pipe:

    http://rss2pdf.com/?url=http%3.....nder%3Drss

  5. Jerome

    Yes, but are they compatible with tubes?

  6. CHEF

    APPARENTLY THIS PRODUCT WAS TOO TECHNICAL FOR ARRINGTON TO INTELLIGENTLY DISCUSS?

    PIPES IS NOT UNLIKE THE “VIGNETTE BUILDER” AND “VIGNETTE BUSINESS INTEGRATION STUDIO” PRODUCTS, BUT FOR THE WEBB.

    BUILDER:
    http://www.vignette.com/dafile.....lderLg.gif

    INTEGRATION: http://www.vignette.com/dafile....._BISLg.gif

    IF YOHOO WAS SMARTS, THEY WOULD ENHANCE THE PRODUCT TO SUPPORT WEB SERVICES: YOU POINT PIPES AT A WSDL, PIPES READS IN THE INPUT PARAMETERS IMPOSED BY THE WSDL, AND PROVIDES AN INTERFACE THAT PERMITS THE USER TO BUILD AN INTERACTIVE EXPERIENCE WIFF IT.

    EXTEND THIS INTERFACE TO QUERY OF JDBC AND OTHER SIMILAR DATA SOURCES.

    COMPLETE THE USER EXPERIENCE BY LETTING USERS PUBBLISH THE RESULTING INTERFACE TO A WIDGET COMPLETE WIFF YAHOOO “PANAMA” ADVERTISING.

    VIGNETTE = PWNED. LOL.

  7. Yohay Elam

    - I wonder if these pipes could be utilized to aggregate pictures from blog posts, rather than showing the whole post or the excerpt.

    - Jerome, what are “tubes”?

  8. Nik Cubrilovic

    Jeremy: Good point but many of the readers here are straight-up consumers, so while I think they should know about pipes they probably won’t use it, possibly on the publishing side though

  9. mathew

    is this what teqlo (jeff nolan, peter rip, et al.) are doing?

  10. eelco

    Very nice! I aggregated some open source planets into one:
    http://pipes.yahoo.com/pipes/7.....s7QZVUMqA/

  11. Oobie Doobie Kanoobi

    I think the lines between what people are calling ‘consumers’ and publishers are blurring. Tools like this make it possible for people with limited tech knowledge to use their imaginations and create something.

    I think this is cool.

  12. Pit Schultz

    what seems to be missing is a “views builder” in the style of netvibes ecosystem. put some webservice actions in between, pepper it with microformats and then let the poweruser play lego with what has been known as the “web page”. web n+1 is crowdsourcing dbdesign to the masses. they are in fact building a new public sphere inside the walls of google and yahoo.

  13. yongfook

    Every now and again a web app is brought out that makes me sit up and say “holy crap!”. This is one of those times.

  14. ...some Drifter

    i’m not that impressed - it’s like a user-generated, aggregated search engine w/ a weird mouse drag feature

  15. Kula bácsi

    Clever product.

  16. Sam Davyson

    Edit a pipe -> Safari crashes.

    Great product.

  17. SearcH EngineS WeB

    The Save and Publish Option has Disappeared

    And the App is running very slow

    Help them if It get a Digg Homepage; this one can’t be DiggMirrored :-p

  18. yongfook

    Sam > works fine for me (Safari 2.0.4). It’s a freshly-launched beta, give it a little slack.

  19. Bill

    This will be of particular interest to me. I’ve been developing http://www.myownsite.us and trying to make it contain more user-driven items such as feed modules for most popular links (based on clicks), most recently clicked links, and displaying the most popular feeds. I’ve been asked to provide a feed based on keywords found in any other feeds that any user has subscribed to.

    I currently have search feeds and have used (but not integrated) sites that bundle feeds together. Is that similar to what Pipes does?

  20. heri

    great marketing tool from Yahoo! also lots of services and informational websites now made possible - and for free.

  21. simon

    A potentially useful product. I think thout that it will only reach its true potential if it becomes possible for external developers to create new pipes, preferabley in the form of a web service. Or to just be able to use existing web services. That combined with some sort of displaying standard could be very interesting.

    Scenario: Take your existing pipe and pass all of the links back to commenters through a reverse DNS lookup (get the ip). Translate the IP’s into latitudes and longitudes, easy enough. And then plot them on a Google Map. Instant map of everyone talking about you. Que evil laugh and thunder clap.

    Simon.

  22. Julio Nobrega

    Amazing. I don’t know how much I will use :p but neverthless it’s an impressive piece of technology.

    I hope sharing pipes was well done, so I can leave to others the work of creating good ones ;)

  23. Bradley Horowitz

    The team will definitely support external web service pipes so that you can freely extend the system. It’s one of the very next things they’ll do ;-)

  24. alastair green

    Very cool

  25. Paul Freet

    Is it just me, or is Pipes down this morning? I cannot acces the site to check it out. Did the Pipes get clogged?

  26. John Griffin

    Yep it’s down for me also.


  27. The Pipes look to be clogged, indeed. It was working for me this AM, so I reckon it got TechCrushed.

  28. xxdesmus

    I am really excited to try this out when I get out of work, thanks for the heads up!

  29. Leandro Ardissone

    Yes, down for me too..

    Can’t wait to try it!

  30. marcel weiss

    I like compete. but since when it is allowed to have animated ads on here?
    distracting..

  31. April

    That’s because Safari sucks Sam. It hates Javascript. ;)

  32. Roger Thompson

    Pipes, hmmm ….. where have I heard that before? Ah Unix Feature for the last 30 years. Simply tarted up with a visual interface. Gary Perlman then at Ohio State showed men something similar 15 years ago. I guess this is the penalty for not reading the literature.

  33. Jeff

    Cool! You guys are sending too much traffic over there…it’s busted!

    I’ll can’t wait to try it out though. For people who aren’t familiar with Linux this is probably a new, cool concept.

    http://averageidea.com

  34. David

    Salesforce also release a open architecture call Apex, which TechCrunch covered a few weeks back. I find this similar, yet far more intuitive — when it works.

  35. pallet jack

    Looks, neat I will try - but it is still down ?

    - I like the idea, but again the terminology is flawed. Pipes is good, but calling (2) things one word is never good, unless its a joke. like -

    - Convince the boss, a “Southern tresspass” is when you tricked someone”

    - then let everyone know it really means, “a slip during a sexual act, into a un expecting place” -

    – Then trick your boss into stating he “Southern’ed trespassed the ugly secretary”- in front of everyone.

    - Ok so I went off topic, but it was funny … (atleast to me) , (must have been there)

    -RB

  36. Matt

    It’s not a truck you can just dump things on.

  37. Yuva

    looks good but very very slow… to load page, etc.. (yes, I do have fast boardband connection but still)

    /Yuva

  38. Yuva

    is it like to Google Reader.. or anything more??

  39. Jerome

    @ Matt, :)

    @ Yohay Elam #5,

    http://www.google.com/search?h.....tnG=Search

  40. Vikas Sah

    It seems to be down right now.

  41. Jerad

    The pipes seemed to be clogged. Maybe if I do a little gambling…

  42. Greg

    Man, they should’ve named it Tubes.

    Looks pretty awesome, that aside. I love seeing quality dev-tools-as-web-app come out of companies, and Yahoo!’s corporate culture seems to support it. Anyone have the odds on whether this idea came out of a Hack Day?

  43. Michael Fomkin

    I was very impressed with yahoo offering a service like this that appeals to a more tech audience. Very good move, gives them back that edge they were missing for so long.

  44. Matt

    Thee pipes are clean!

  45. Jonathan Trevor

    Yup, as someone said, our pipes are indeed clogged at the moment :-(

    We’re working it…

  46. Jay R.

    Has anyone had trouble loading? I get “system error cannot load” or an infinte loading symbol. Im running firefox, xp, and on university fiber. Thanks.

  47. icebin

    Cool.I like it.

  48. Sundar Krishnamurthy

    Long time ago, in a decade far away (read 80’s and early 90’s), there was a company in the Silicon Valley called Metaphor Cmputer Systems (which was later acquired by IBM). It got lost once IBM took over.

    Founded by ex-Xerox PARC researcher David Liddle, Metaphor had a tool very similar to Pipes. Based on Xerox Star, it had some advanced features that made their way later into Windows and Mac. Nostalgia buffs, see http://patricklogan.blogspot.c.....stems.html for more info; for the latest incarnation, go here — http://www.meta5.com/M5Product/Product.asp.

    This tool can visually connect a variety of data sources including spreadsheets, word process documents, databases, etc. in addition to API for converting any application to be a data source or a sink.

    It is an elegant concept for quickly prototyping data conversion/processing applications and it is nice to see that Yahoo is doing it.

  49. babagugu

    I like the new error message on the sample yahoo pipes posted above. The message says:

    Our Pipes are clogged! We’ve called the plumbers!

    At least the developers have a sense of humor. :)

  50. Jeff

    Our Pipes are clogged! We’ve called the plumbers!

    Oh Yahoo! You’re so Google-Like.

  51. Alaskan Carnivor

    The Yahoo pipes are clogged for me.

    They have been for the past 15 minutes….

    And it’s not just the TechCrunch pipe.. The while thing .. http://pipes.yahoo.com

    Tubes, Pipes…. Whatever..

    Nothing more than buzzwords when either are plugged..

  52. Sean

    I clogged my pipes … those refried beans get me every time!

    “Hey Yahoo, send your plumbers to my house when you’re done with them!”

  53. Ty Graham

    What is the difference between seeksift.com, dappit.com and Yahoo pipes? I would really love to hear a detailed comparison.

  54. Zbigniew Lukasiak

    This is like a mechanism for my Social Routing idea (http://zby.aster.net.pl/kwiki/index.cgi?SocialRouting) - i.e. a system where you can not only send items for other people - but you can also build mechanisms automatically sending them some items.

  55. Yakito

    I can’t belive Yahoo can’t handle the traffic. Site is down

  56. belmar cam

    Our Pipes are clogged! We’ve called the plumbers!

    what is yahoo smoking?

  57. Alfred Toh from Payscroll.com

    “Our Pipes are clogged! We’ve called the plumbers!”

    That is funny as hell..

  58. Jrome

    I agree that the name is medieval, but I like it. Sounds like something a rare earth magnet could stick to or something an average Joe might understand.

  59. Harsha

    This thing sure does look like it has a lot of potential. Why did no one get a wind of it before it was released?

    “Our Pipes are clogged! We’ve called the plumbers!” - This was the meassage I was repeatedly getting when I wanted to see for myself.

  60. Bill Patrick

    Seems mixing up Internet content is becoming all the rage. First this Yahoo Pipes thing, then I just found this video on Digg, of some other browser-based mashup tool. Looks like Greasemonkey on steroids!

  61. Andrew Moyer

    Pipes!? Everybody knows the internet consists of tubes!

  62. Matthew Berman

    too bad its down already

  63. The Biblioholic

    I guess they need to call Roto-Router [sic]

  64. murphtron

    Same question as #9 - how is this different from Teqlo? And will Teqlo survive, as a result? ;-)

  65. xxdesmus

    Too bad the pipes burst already, I was looking forward to trying this out.

  66. RT

    Damn,

    That is one long ass explanation of what a product is and does. I’ll need to set aside some time tomorrow to read it all. First instinct - Taking that long to explain what a product does = no good.

  67. carmen

    xxdesmus: write Pipes.js so we don’t have to wait. i give that 2 weeks to appear plus a torrent-tracker type site to exchange recipes

  68. mike

    How is this different than net vibes???

  69. Camilo

    Looks like the pipes is down for a while ( “over-demand” of the service). Looks like netvibes, google home page ,… to me

  70. lucidcheese

    My earlier attempt at combining Amazon webservices with the world of UNIX

    Amazon CLI: http://lucidcheese.com/ACLI

  71. Anonymous

    This is again, an attempt for non OOP languages to attempt to create a framework in which .NET already has. Just save yourself the trouble and switch to C#

  72. Anonymous

    I meant non .NET languages, not OOP! Java sucks, screw that. Move to a luxury car that you can also tune as needed! .NET with all the nice works..less hardaches.

  73. saumendra

    YAhoo pipes, is one of the most awaited product in web 2.0 family. Earlier products like the ning, were more of a blueprint, on which yahoo pipe is framed. But the way its been structured , is worth watching .

  74. Emmanuel

    Does Pipes make an easier environment than quomodo (beta.quomodo.com)?

  75. AD

    I am scratching my head - why would I want to use this service? Can you explain in simple terms what it’s for - why people need it and stuff? All this stuff about unix, pipes and dragging-and-dropping stuff makes no sense - why would I want to spend a few minutes dropping content and connecting them with pipes for?

    I really don’t understand what this is for.

  76. Julio Nobrega

    @75

    AD,

    Imagine being able of taking any feed, performing operations on them (sorting, filtering by word, mixing with other feeds, searching) and outputting another feed.

    Connecting commands on the Unix world is done with the pipe operator, “|”. But Yahoo Pipes is a graphical tool, thus the drag and drop.

    Why do people need it? To search, transform and create feeds, that’s why… :p

  77. Josh Windsor

    Looks like more than just pipes being clogged. I can’t access anything on Yahoo. Great technology!

  78. Jeff Lindsay

    I particularly liked Tim O’Reilly’s response to this, and I have a very similar sentiment, but I think this is a great time to point out that “pipes for the web” won’t be fully realized with feeds.

    Pipes for the web has an amazing potential. It’s the idea of achieving what the web originally did for content, but for logic… a higher level platform of interconnectedness.

    This is a great step in this direction, especially with the use of the graphical programming environment… but feeds are limiting the potential.

    The problem with feeds is that they’re weren’t designed for server to server communication, they were designed for client software consumption. They are not the ideal transport mechanism for software to software communication and the name “feed” is slightly misleading.

    First of all, they’re not real-time. They’re close, but even then they’re not very bandwidth efficient. I don’t see how we can call them feeds when we have to poll them!

    Instead, we need hooks. Web hooks. We need applications to let their users specify a URL to POST to on events with data. Simple form encoded posts. They’re easier to implement than RSS and WAY easier than an API. However these could almost entirely replace some API’s.

    It’s not the short term obvious benefits they provide, but the long-term benefits. The new ecology that will emerge that would include tools like this. But they’d be more powerful tools. We’d get things like Automator for the web, but for real.

    The problem with comparing the web with Unix for pipes is that web apps have way more *asynchronous* outputs than STDOUT or STDERR. But to make this pipes thing work, we need to expose more outputs.

    With web hooks we can transparently share data between web applications. Not necessarily driven by vendor integration, but by *users that wanted it to happen*. The vendors only have to let them. With hooks.

  79. David Mackey

    Looks pretty interesting. Wish I had more time to look at this.

  80. The Muso

    I’m too late on this for a response I know, but CAN ANYONE GET THE PUBDATE SORT TO WORK? Nik yours doesn’t work either, it sorts alphabetically instead of by date published and if you take out sorting by title, it just displays in batches according to whichever feeds are retrieved first. A great product, really annoying flaw. which is going to stop me using it.

  81. Matt

    feedGod has had a simplified version of Yahoo’s “Pipes” available for a while now. Not as many options as Yahoo’s but it’s much easier to use.

  82. UsefulVideo

    Indeed, the main thing that Yahoo Pipes has done is that it’s made a difficult programming chore more accessible to mainstream users. Just as so many programming languages make assembly language more accessible. However, there is still quite a learning curve for the average end-user. I have created some video tutorials to help lessen this curve. With your permission, here is the link: http://usefulvideo.blogspot.co.....rials.html

  83. John Doe

    I do not like the name feedGOD.com, sounds very disrespecting.

  84. Brad Gessler

    I started to use Yahoo Pipes to create my very own “Life Feed’. I combined my blog posts, diggs, and flickr photos into one feed so that everybody can follow my online life. I put a tutorial online on my blog at http://www.bradgessler.com/200.....pipes-app/