Yahoo Pipes End-Of-life Announcement | Hacker News Reader

Yahoo Pipes End-Of-life Announcement

pipes.yqlblog.net

361 pointsmichaelhoffman177 comments

Pipes was the first service I ever owned from an ops perspective (circa 2008). We had two clusters, one on each coast. I thought we had capacity to take one offline to do a full upgrade, turns out we didn't.

By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.

Fun times.

Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.

Business model :)

Just to be clear, I had not idea about that until now.

Did your remove the affiliate ID?

We chuckled and moved on.

> had his affiliate id in all of those links

Wow that must have added up to quite the haul

I wonder if they had put in a Yahoo! owned affiliate link, if the site would not still be operating (and self-funding) today...

Haha, I loved that search results pipe! The product was amazing at the time. It's really too bad that Yahoo didn't try to make something more out of it.

I worked on Pipes briefly, shortly after it launched. It was really two distinct products rolled into one. The visual interface was groundbreaking and stunning-I think half my team was hired on the strength of that demo. No one could believe Javascript could do that.

The backend was an extremely useful tool for munging RSS feeds. With any kind of support, or even benign neglect, the product would have been successful. It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

Hats off to psadri and the other Pipes creators for a really stellar piece of work.

Thanks :)

Pipes's interface is primary the work of Jonathan Trevor, with lots of help from Daniel Raffel, Ed Ho and others.

> It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

I feel like this unique blend will be Yahoo's lasting contribution to the field of management theory.

DEC/Compaq/HP is really hard to top in the area of mismanagement.

Don't forget Nokia. Going from #1 in a market to doing so badly that you exit the market--and all in the span of 3 years--is something to behold.

good one. it's what one famous blogger calls the 'elop effect': http://communities-dominate.blogs.com/brands/2011/08/coining...

Ahh, what a terrible CEO. When the only two possibilities to explain your tenure are 1) a lying co-conspirator or 2) mind-blowingly incompetent--then you've truly done a bang-up job.

By the way: many people in the past, including here on HN says that Elop was right, and that the reason Nokia died was 100% apple+google fault.

Here in Brazil Elop effect was interesting: it was VERY obvious, yet noone here knows who the hell Elop is.

To put it simple: When Elop released that stupid memo, Nokia had 67% of Brazil smartphone market share, and the share was still rising, and Nokia utterly dominated in the non-smartphone market too. Right after Elop memo, the first thing that happened is that suddenly the communities died (although Nokia store DID suck, there was at least in Brazil a vibrant freeware app community that shared stuff in forums), then just some months later you could not find phones for sale, even new models, because the import companies DID heard of Elop memo, and decided to not risk importing the phones, even if shopkeepers still wanted it.

The end result was funny: Elop nuking of Symbian was so hard, that in Brazil other smartphones could not keep up for a while, and we temporarily had a decrease in smartphone use, and a increase in feature phone use, and that increase came mostly from truly crap chinese companies, because not even Nokia feature phones could be found in stores anymore.

It wasn't Elop that sank Nokia, it was Olli-Pekka Kallasvuo.

By the time Elop was announced as CEO in late 2010, Kallasvuo had already guaranteed Nokia was toast.

Just go back to the way Nokia was behaving, and what they were saying in 2007/2008 for easy proof of that.

"Nokia CEO calls Apple iPhone ‘niche product’"

http://macdailynews.com/2008/04/17/nokia_shares_slammed_in_w...

http://www.informationweek.com/mobile/nokia-ceo-disses-iphon...?

http://www.forbes.com/2007/02/12/nokia-kallasvuo-iphone-face...

http://www.forbes.com/forbes/2007/1112/048.html

> Elop was a director of consulting for Lotus Development Corporation before becoming CIO for Boston Chicken in 1992,[3][7] which filed for Chapter 11 bankruptcy in 1998

> In the same year, he joined Macromedia's Web/IT department[7] [...] from January 2005[10] for three months before their acquisition by Adobe Systems

> He was then president of worldwide field operations at Adobe, tendering his resignation in June 2006 and leaving in December,[12] after which he was the COO of Juniper Networks for exactly one year from January 2007 – 2008

This is not the CV of a man that I would hire for anything other than running a going concern into the ground.

i think nokia is more about lack of executive vision/market knowledge. The other examples are more about middle management, besides the executive lack of vision and bean counting.

I feel like Atari might be the grandfather of all this.

And Commodore would have to get an honorable mention

Is Nokia's example really good ? Nokia remain one of the largest companies in telecom hardware provider and handsets was always a side business for them. They successfully sold it to microsoft.

I'm still upset by the fact that MeeGo/Maemo was killed, and Nokia sold to Microsoft. The N9 was still the best phone I've ever used, and it's UX ideas are still the simplest and most effective way of using a phone, and yet no one else has implemented it.

Jolla (https://jolla.com) was founded by people involved in MeeGo development. While their phone got a lukewarm response, I've heard many good things about their upcoming tablet and have ordered one. As you used MeeGo, how does Sailfish OS appear to compare with it?

Sailfish unfortunately does not have all of the UI things I'm talking about -- the "three screen with edge gestures", which is a shame :( They couldn't, because the N9's UI was never open sourced despite the core being so.

Agreed

It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.

I would love to read a blog post about how exactly it was mismanaged. It would be great for others to learn lessons from the mistakes.

The first rule of Mismanagement Club is you never talk about how you got into Mismanagement Club.

Love the reference!

Incorrect. The first rule of mismanagement club is to never know exactly how you got into mismanagement club; but by god you are going to do a bang up job of it now you are in!

Get a job at Twitter and you can see this in action. It's pretty epic.

I think most/many people could reasonably say this about their company; I currently work at Google and feel quite the same. I'm not convinced that either Google or Twitter[1] qualify as a particularly spectacular example of mismanagement though.

[1] I don't have first-hand experience but one of my best friends work there and we regularly vent to each other about work complaints.

I wrote a simple Wordpress plugin that works with RSS feeds. In addition to aggregating fees, Pipes has been a go-to solution when people report feeds that are malformed, have some weird characters or anything that chokes the Wordpress RSS parser (simplepie). People would run them through pipes, which seemed to handle anything and output clean RSS.

I'm sad to see it go and it'll be missed by quite a few people who use my plugin.

maybe www.busit.com can replace it for that matter

It was really great to work with you. During the post-launch chaos, it was life saver that you didn't have your hair on fire like the rest of us :)

Jonathan Trevor's Javascript was out of this world amazing for the time.

I'll echo those sentiments, the UI for that time certainly had a wow factor.

> With any kind of support, or even benign neglect, the product would have been successful.

Look at IFTTT services now and it's easy to realize that pipes even in its original incarnation could have been so much more given its brain dead UI ease.

It's a shame that Yahoo is shutting this down rather than pivoting it as an IFTTT or similar. I'd wager that it's lack of ownership that makes it hard to pivot. I also don't recall it having a premium tier which has got to weigh down the resources--talent always gravitates toward revenue centers it seems.

> talent always gravitates toward revenue centers it seems.

Ie the 'new management can't fire my entire division for being unprofitable' center. Seems like a smart move for the people who wrote Pipes but never found a business model for it.

pipes creator here: sad to see pipes shutdown. i am surprised it lasted this long -- it was abandoned years ago. on the positive side, it is nice to see it mentioned in the same sentence as y! maps which used to be a big deal back in the day.

It was really freaking awesome. Thank you!

Please suggest alternate. You built a good thing. RSS overall is becoming history.

IFTTT and Zapier are spiritual successors to Pipes, although their focus is on APIs vs feeds.

Maybe start a new, open source version of Pipes?

It really would be oh so nice to have something that can ingest RSS and Tweets and then apply filters to them while visualizing the logic through a graphical interface. Maybe even some fuzzy machine learning to use in building filters. Some more advanced dupe technology would go a long way. People would be building their own techmemes in no time. You could build ranking systems that award bonus points if something like longform or techmeme mention something.

Maybe im dreaming about a free version of Percolate, which I have heard is really cool.

For what it's worth, that is exactly one of the things I am building. Glad to read I am not the only one needing it.

One of our open source projects[1], lets you consume RSS, Email, and Tweets, with more "stuff" coming, and apply filters and set triggers, etc. It's still pretty raw, and we don't have the really awesome visual interface for laying things out, but that sort of thing is something we hope to get to eventually. One other neat thing we do is pipe content through Apache Stanbol to do semantic entity extraction, and build up a knowledgebase as content is consumed.

[1]: https://github.com/fogbeam/Neddick/

This is exactly what we are doing at http://www.feedsapi.com , we don't use a graphical UI for the logic but keywords and RegEx to create filters... Many folks are already building their own techmeme with our API, it's however not a free service. I'm not sure if this fully fits your requirements but you should take a look.

Checkout https://github.com/olviko/RssPercolator

Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.

No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.

Founder of Enginuity Analytics here. You can ingest RSS and Tweets into our platform where we apply machine learning to provide social data, sentiment analysis, rankings, filters and demographic analysis of all the content for 27 languages. And easily export the data to Excel and more. http://theenginuity.com

Try Huginn. I found it very easy to setup and work with. https://github.com/cantino/huginn

Instead of closing it down, releasing the source and giving it to the community could breath live into it. I remember being blown away by it, the fact that it was neglected and just kept on says a lot about it's good design.

This would be the right solution.

Check out http://www.feedsapi.com , we do almost everything with rss feeds...

Checkout https://github.com/olviko/RssPercolator.

RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.

- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)

This is an amazing product. Have you tried talking to VCs or anyone about someonr adopting it somehow?

I wonder if it would be possible to build something similar using IPV6/some kind of P2P/STUN/TURN/UDP hole punching/swarm/ethereum / any way to make a distributed system that does the same thing. Or maybe just volunteer running Node on their own VPSs.

Something like Pipes running on sandstorm.io would be nice.

Why did it fail, do you think? I thought it was an outstanding product in terms of both functionality and UI, but it never seemed to gain traction.

Several reasons I can think of. For one, no one at Yahoo seemed to understand what Pipes was for (part of a wider problem where Yahoo didn't seem to know what the company was for).

For another, Yahoo legal prevented us from doing any kind of long-term storage of RSS feeds. That made the service far less useful than it could have been.

Finally, they didn't give it any resources at all. Management was too preoccupied with the failed Brickhouse project (internal startup incubator) and the host of failed ideas that it spawned.

no one at Yahoo seemed to understand what Pipes was for

What was Pipes for?

Filtering and triggering without the annoyance of having to write code in some syntax-of-the-month or dredge through poorly written documentation for half-ass APIs.

> legal prevented us from doing any kind of long-term storage of RSS feeds

Wait, is there something legally wrong about storing RSS feed data?

If you are redistributing, hosting or using it commercially you have to have the agreement of the author of the content. RSS just gives you access in another way than HTML, but it does not circumvent copyright.

Thank you for the great work.

Let me just tell you and anyone that worked on the team, thanks so much for it. You saved me hours of work and let me do it really damn well. This is a sad, sad day. I used Pipes extensively and ironically enough at Yahoo for some features/prototypes on News and Search. It was such an elegant and powerful app, I really wish Yahoo had given it more love.

Thanks so much for you and your team's work on Pipes - I'd be lying if I said it didn't inspire many parts of Zapier!

Same here... Pipes has been an inspiration to use with a lot of the stuff we've been doing as well.

Pipes has long been a sort of monument to the faith of a bygone age -- the first flush of Web 2.0 optimism, when standard formats and open APIs were going to let us mix and match and mash up services of all kinds at will.

That faith is dead now, of course, scoured from the earth by walled gardens and VC money. But like Catholics in Elizabethan England, some of us quietly tend our secret shrines and pray for its return.

(Hopefully that will work out better for us than it did for them.)

Yep, I was trying to figure out how to say what you just said, but you did it better.

Thank you for your comment, for a (long) while I thought that I was the only one remembering those times and especially the ideals people were bringing up then. Sometime in 2005 I even copy-pasted this quote on my blog, high in believing that the future will belong to open data and open information:

> Bosworth advocated an open model for data.(...) Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume. He also pointed out that "complex things tend to break and simple things tend to work." Google has the simplest query language in the world. There is no structure and no syntax. (my note: at that time Bosworth was VP of Product Management at Google)

That is all dead now.

Perfectly worded.

Love your analogy, sure made me laugh (visualising a geeky dev like myself tending a weathered gnarly monument in a secluded English glade... in real life, that is, not in an RPG).

However, why such pessimism? I hardly think that standard formats and open APIs are dead. Sure, there are some nasty blights upon cyberspace in this regard (ahem, Facebook, Apple), but other players are still keeping the dream alive to some extent (e.g. Google, Yahoo).

I'd count google mostly as an example of deprecating or limiting useful interfaces. Yes, you can feed them stuff in "open" formats, but getting stuff out is harder. First example that comes to mind is youtube APIs, which offered easier access to stuff. Now try to get a feed of someones videos or similar...

But it is possible that it's confirmation bias + uncommon sample on my part.

Now try to get a feed of someones videos or similar...

What do you mean? Channels have an element linking to an RSS feed. In fact, just pasting a channel URL to an RSS reader should work.

Example of a channel feed: https://www.youtube.com/feeds/videos.xml?channel_id=UCr4soU_...

.... true. So they deprecated the methods for it in the API, but it is right there on the page. Strange.

Yeah... for Google / YouTube, I blame OAuth for most of the difficulty in accessing data (while retaining one's sanity) these days. Once you've waded through the quagmire of access tokens, auth libraries (in different languages), account permissions, etc... making the actual API calls isn't so hard.

Hey Pasha, thanks for your pioneering work!

We (Webflow) are actually considering building a Pipes alternative in the near future, and I'd love to sit down with you and brainstorm some ideas. If you'd be willing to make some time to see a very rough demo, I'd love to get in touch - my email is in my profile.

Thanks Pasha for letting me be a part of your brilliant idea.

Thanks for creating something so awesome. I have relied on it heavily for years to clean up dodgy feeds.

Seriously, I think the only Yahoo products I use are Pipes, Groups and Flickr. And Flickr doesn't really count.

In an alternate universe not too dissimilar from our own, Pipes is an independent startup and this announcement is Yahoo buying it for 8-9 figures

There was IFTTT but looking at their page now it seems like maybe they have pivoted, at least I can't tell that it works like it used to: https://ifttt.com/

If you sign in all the previous functionality is still there. Seems their landing page is only advertising their new "Do" apps (which are a one-button extension of the old IFTTT recipes). Installed it, but never used them much.

Many see tools like http://Zapier.com or http://elastic.io (disclaimer - I'm a founder) as alternative to Pipes and IFFT.

But all of them seem to miss the focus on RSS, or even just the ability to manipulate feeds.

Zapier is the only reliable tool I found so far that can still import RSS to facebook pages, it tried it with pipes and it had issues with non-standard fields.

The whole internet misses the previous focus on RSS...

It's amazing how much a shift of management focus can change things... Google embraces RSS, develops the premiere online RSS reader, so good that even stand alone readers are largely abandoned... then google nukes RSS support from Chrome, and shutters Reader. And the world is worse off for it.

It definitely inspires a "never forget" kind of mindset, such that I don't trust relying on a SaaS that I don't have a personal exit strategy for.

Twitter took RSS as a concept and humanized it

RSS is an open format, Twitter is a walled garden.

If Twitter is making it more human, I'd rather be ruled by the robot army...

IFTTT works well for feeds, also had good success with Huginn in the past for feeds.

IFTTT still does the same things it always has.

I liked the concept and probably didn't give it a fair shake, but it flooded me with notifications and I just didn't want to take the time to tweak it to my needs. Its one of those apps you need a good tutorial (from someone other than the creators) to walk you through useful functionality and setup.

Wow, this is a jumble of I-don't-know-what:

https://ifttt.com/products/do/button

Their site aesthetic is trying to be Nest-y, but the UX does not communicate simplicity at all.

Too true.

The only thing Pipes would need is a .io domain and it could be dressed up as the Next Cool Thing which LargeCorp is desperate to get its hands on to prove it's still cool.

Yahoo pipes was obviously influenced by UNIX pipes and as such is one of the best demonstrations of flow based programming to date. I really loved Yahoo pipes and it influenced the way I think.

Pipes was a basic web "agent". It made basic programming available to the everyman much like HyperCard. Perhaps is just needed a runtime UI that matched the excellence of its design time interface.

Yahoo is focussing on mobile yet here they had this custom agent building tool that could easily be re-purposed to mobile to make a killer platform for Yahoo users.

Hopefully they will opensource the first generation perl version.

Some of my favourite flow based programming links follow.

Surprised nobody has mentioned noflow node.js workflows:

  https://github.com/noflo/noflo

Also this (untested) will convert your yahoo pipes into Node.js

  https://github.com/neyric/pipes2js

and this one (also untested) will do the same in python:

  https://github.com/ggaughan/pipe2py

There is also python pypes which has a yahoo pipes like frontend with a stackless based backend.

  http://pyvideo.org/video/400/pycon-2011--large-scale-data-conditioning--amp--p
  https://github.com/fullscale/pypes

Other python flow based programming tools

  https://wiki.python.org/moin/FlowBasedProgramming
  http://www.kamaelia.org/Home.html (BBC research)
  http://www.ruffus.org.uk/index.html

Wireit is a javascript pipes like frontend

  http://neyric.github.io/wireit/docs/

As you can see I loved pipes and I can

Correct and thanks for those links.

Pipes was influenced by unix pipes (hence the name). The UI was influenced by National Instrument's LabView and Apple's Automator.

I wish Pipes was launched in the age of containers (e.g. Docker). We had the idea of one click deployment to what we today call a container.

I really hope there's a new service that can pick up where Pipes left off, it was a huge shame how it fizzled out at yahoo.

Paul Haberli made ConMan on SGI workstations:

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.78.5...

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78....

ConMan: A Visual Programming Language for Interactive Graphics (1988)

Abstract

Traditionally, interactive applications have been difficult to build, modify and extend. These integrated applications provide bounded functionality, have a single thread of control and a fixed user interface that must anticipate every-thing the user will need. Current workstations allow several processes to share the screen. With proper communication between processes, it is possible to escape previous models for application development and evolution. ConMan is a high-level visual language we use on an IRIS workstation that lets users dynamically build and modify graphics applications. To do this, a system designer disintegrates complex applications into modular components. By interactively connecting simple components, the user constructs a complete graphics application that matches the needs of a task. A connection manager controls the flow of data between individual components. As a result, we replace the usual user-machine dialog with a dynamic live performance that is orchestrated by the user.

Most of what Pipes did can be done locally, in a library. FeedReader for Windows did this.[1] Unfortunately, they recently announced "Get ready for the brand new FeedReader! ... Check our site later this March for a new amazing web service, featuring industry-best RSS search and analysis.", which means it's becoming "cloud based" and probably has ads or worse. "rss.com" already went that way; they charge $6/month to read RSS feeds.[2]

Following RSS feeds on a continuing basis takes a lot of RSS polls. Most RSS feeds do not implement RSS in a way that allows getting only new items reliably.[3] The RSS "etag" mechanism is not reliable. Some sites with multiple servers and a load balancer have different etag values on each server. The "guid" field sometimes changes when the content hasn't changed. My experience is that nothing short of full text comparison eliminates duplicates properly. I wrote an RSS reader which does a MD5 of the text of each incoming message to throw out duplicates. Presumably the "pipes" system did something similar.

If RSS feed servers complied with the standard, there'd be less need for feed aggregation services.

[1] http://feedreader.com/ [2] https://www.rss.com/ [3] http://www.詹姆斯.com/blog/2006/08/rss-dup-detection

I think your first paragraph speaks to a real tension in the world of software: web applications have enormous discoverability, immediacy, upgradeability, and (often) social collaboration advantages over native applications, but they have ongoing costs proportional to their popularity, so it's harder to run them without ulterior financial motives.

Aw man - that is sad. We at Zapier took great inspiration from Pipes, it was such a cool and powerful product! It was a shame to see it languish.

I'd sure love to pick the brain of anyone formerly involved in the project - it seems like there is a lot we could learn from the trail it blazed. My email in profile, obligatory beers/coffee/etc. offer.

You guys are fantastic. I use Zapier for several of my organizations and companies, and I cannot understate how incredible of a service you guys have created. <3 for days.

My only wish would be for a paid tier between 100 and 3000. If I went hog-wild and did everything I want to do, I'd probably use 200-300 a month. I would love to pay for that. But there's some loss-aversion psychology happening that makes me not want to pay for 3000 if I know I'll only use 300.

I'd give you my cc number right now if you did a 1000 zaps at $10/m.

That being said - you know your business better than I do. If it would hurt conversions of $20/m more than it would help, I get that. :)

<3 you guys though.

Thanks for this Zach, we keep re-evaluating and running pricing tests, and this plays into that a bit. Reach out anytime if you need help!

I actually had the same query last year when I pushed a place I work at to use a combination of Yahoo! Pipes and Zapier for our Facebook postings. I asked for a smaller plan (we only need about 600-700 zaps/month at the most) and was just instructed by their support to utilize their reward program[0] which offered an extra 50-100 zaps/month based on what you 'complete,' and it included things like (IIRC) liking them on Facebook or following them on Twitter.

It seems like that program has since been limited, however, as it now redirects to a referral page[1] only. A combination of a couple referrals around the office and performing the required tasks gave us about ~650 monthly "zaps" at the time and we haven't needed more than that so far. Still, I would have happily paid $5/month or something to avoid the hassle (and be able to pay them for their wonderful product).

[0] https://zapier.com/rewards/

[1] https://zapier.com/referral/

Yahoo Pipes was the backbone for ProPublica's first news web application in 2009: http://www.propublica.org/article/changetracker-howto

> ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via e-mail, Twitter, or RSS. But ChangeTracker is not a piece of software. It’s the output of a series of powerful and mostly free Web-based tools, lovingly connected over the Internet. Here’s how to do it yourself so you can track changes on any Web site on the Internets.

I used Yahoo Pipes to find my apartment.

One pipe would grab data from Craigslist, strip out irrelevant items, and send me a text if anything new and interesting appeared. Ditto for Westside Rentals.

This combination worked great. If a lame apartment appeared in CL, I'd edit my Pipes regex to strip the same thing out of WR. In this way I'd get only texts for awesome places.

Vaya con dios, Yahoo Pipes!

Ah, I did this too! I'd forgotten about it. As I was looking, I'd continually update it to block out streets I wasn't interested in, then eventually I was only getting notifications for exactly where I wanted for exactly how much.

I also used it with Freecycle and got a fridge-freezer! (you had to be quick to get them)

I was also using it exactly like this until recently! I sent my roommate the pipe and we'd each watch/browse it for new places. It saved us a ton of effort.

I've used this pipe as my Hacker News RSS feed for the last 5 years, which flips the link structure so the main link for each entry is to the comments.

http://pipes.yahoo.com/pipes/pipe.info?_id=0e19519adf0515251...

Thanks to whoever made it. I guess I'll have to find a way to do it myself now.

The main one I made was to switch the actual link for the Reddit RSS feed to be the main link, set the true article title as the item title, use the submitted title as the item body and add a link to the link's domain in the bottom.

http://feeds.feedburner.com/RedditArticles

And another where I put the pictures inline (though the 3rd party API I was using isn't handling things too well anymore)

http://feeds.feedburner.com/RedditPicturesWithPictures

I still have 146 subscribers to keep happy, though Feedburner should make any transition transparent. I'm disappointed it's going. Thank you to those who made it!

I used Pipes as the basis of data mashup classes, with non-developers students that were mostly afraid/bored of anything related to programming. Using Pipes, in just a few hours they would make small, custom data apps. A typical example would be a student building a pipe gathering the weather from one source and trying to fetch the relevant clothes from a retailer with pictures and links. I had a student using the location entity extraction to geocode his favorite street art blog that cited cities in the text but didn't use rssgeo, and then output the augmented feed to a map view.

It was an eye opener for some: they could get, transform and repurpose data, and it could be fun.

Pipes wasn't without its flaws: V1 was buggy and gave us a lot of trouble when 30 students were trying to use it from the same IP, the interface was plagued by the usual issues of visual interfaces (clutter…). Some things were odd, some a little too hard for what they achieved and other were magically easy!

But at a moment when every tech company is saying that "learning to code" is important, it's sad to see a tool that had real educational value disappear. It was a really effective tool for non-developers to learn about data markup standards, to think in term of data flows, to get introduced to the idea of a data or web API. For this particular use in education I'm not sure there is a replacement.

Who is working on a pipes replacement and where can I sign-up? Maybe this will be like when google killed Reader and RSS replacements became a hot area...

Have you tried IFTTT? Not sure what your usecase is, but that might do the job.

IFTTT doesn't let me basically reformat an RSS feed and then see it in my RSS reader. Most/all of my use of pipes was "go scrape this thing and turn it into RSS, filtering out stuff I don't want to see."

It was great for that - I'm unaware of anything that is even trying to replace it.

I've used this beforem but it's been a while http://createfeed.fivefilters.org/

THIS is exactly what http://www.feedsapi.com does, it's a paid service with a 2 weeks free trial.

    Who is working on a pipes replacement

I'm curious if anyone has suggestions about solving a piece of that:

What would be the best toolset to recreate the Pipes UI?

It's pretty slick, and I could see the underlying UI being useful for modeling a variety of things...

Huginn fits a lot of Pipes' usecases: https://github.com/cantino/huginn

I was looking into doing a Pipes 2.0 sort of project. If you live in the bay area i'd love to get coffee and chat about it. There are a number of the pieces to pipes that are opensource, but none of them are strung together.

I also thought that pipes queries were amazing, but slow, so it was hard to use them as part of an application.

I'm too far away for coffee, but my email address is in my profile.

We're working on something that covers a lot of the same ground, although it's not exactly the same thing. It's also not very polished right now, but we definitely hope to get to that level eventually. It's all open source, Apache Licensed, and built using Grails. We don't have YQL, but we extract entities from the content and store it in a triplestore that can be queried using SPARQL, and eventually the SPARQL support will be woven more completely through things.

https://github.com/fogbeam/Neddick

Shameless relevant plug: http://www.feedsapi.com might be an option, it'sit's not exactly a replacementbfor pipes but it does a lot of what pipes did and is a work in progress

Just a shout out to the awesome Pipes community and some people that really helped it along the way. All the original creators (pasha, JT, ed, daniel), community members and team: hapdaniel, dawnfoster, kentbrewster, mirek, spullara, janluehe, dspark, kevink, brettp, lolo, laurencecoates, earth2marsh, psychemedia, davglass, ssaine, sadaf, ido, nagesh, ameya. I know i'm missing tons more, but just top of my head. You are all awesome.

Paul, thanks for keeping it running for so long. Pipes was the prime mover behind "suddenly, everything is mashable."

Thanks for all of your work. Wouldn't have been the same without you.

Thanks guys, it was such an exciting product to work with, create with and help all the community members. I learned the most from this product.

Oh man, I use 3 or 4 pipes to filter various RSS feeds. Ugh. Good suggestions for replacements (especially those that don't require my own server) would be much appreciated.

Are they also shutting down YQL? I think YQL queries can do everything pipes can, you just have to write little scripts to do it instead of the UI.

Still has the Yahoo problem, but it might be the easiest short term.

  Still has the Yahoo problem, but it might be the easiest short term.

The Yahoo problem being that it might be shutdown in the near future?

Maybe they're focusing on YQL instead, if it does offer similar functionality.

My understanding is that YQL is being used as the middleware between services internally. Will probably not be shut down, but they could shut off the publicly accessible one.

One thing not mentioned in this thread is that Pipes 2.0 was a rewrite from the original Pipes which was written in Perl. We ported it to the same engine that powers YQL.

As for why shut down Pipes if it shares infrastructure with YQL? Probably because usage is mostly RSS crawlers that no one looks at anymore.

YQL queries can do everything pipes can, you just have to write little scripts to do it instead of the UI

Argh! The UI was the innovative part. The whole point of something like this is that you don't have to learn a bunch of syntax - when you add a module, it tells you what sort of data it needs to know (such as a URL or...) and when you connect modules together it tells you what sort of inputs and outputs are available. This allows you to focus on the actual problem domain instead of the morphology of the query language, which is not important to the user.

I keep encountering this attitude among programmers - 'I type code all day, so why don't you want to do the same thing?' If people wanted to type stuff all the time the command line would still rule, but the majority of people prefer graphic user interfaces because they don't want to write code, they want to make selections from available options and use the computer to take care of the plumbing. This is a Good Thing. It's easier, more fun, and yields greater productivity, for the same reason that handing someone a box of Lego is better than handing them a few blocks of wood and a set of carving tools. The existence of Lego hasn't led to a collapse of sculpture or mechanical engineering. On the contrary, the simplicity, consistency, and interoperability of the different components has made a massive creative, educational, and commercial success.

I don't mean this as a hit on you personally, but I find comments like the above frustrating. Imagine you went into a police station to report the disappearance of your bicycle and the person behind the desk shrugged and said 'you're still able to run around on your legs, what's the problem? Your bicycle may have allowed you to go faster down the street but it's not like you could ride it up and down stairs so you should be happy with walking on your feet.' The problem is the loss of a tool that offered substantial efficiencies on some common tasks even though it wasn't ideal for every situation. You'd probably be equally pissed off if your favorite high level language went away and you were told you could just go back to writing assembler. I personally love doing things in assembler because it makes me feel smart but that doesn't mean it's better than high level languages.

Yeah, I wasn't suggesting that YQL was equivalent to Pipes, just that it might be able to replace the functionality. Which is why I made sure to mention the significant difference...

I get the impression that YQL is suffering from the same kind of neglect Pipes did. It was down multiple times a week for me, and I had to move to other solutions.

Suggestion for your use case: http://www.feedsapi.com , supports keywords and RegEx filtering among other things.

Thanks. Guess I should have specified "preferably free" too :)

While working for YDN I was the one who spilled the beans about the top-secret Pipes _callback parameter, which turned your JSON reply into JavaScript. The service fell over instantly, of course, but far from being upset with me, the team was thrilled with the exposure. Viva Pipes!

Any word on whether YQL will live or die?

I use Yahoo Pipes to get around Netflix's walled garden.

Netflix does not believe in RSS, even though their Influencer content is not behind a subwall. I use Pipes to parse Netflix's json content and then turn it into a feed. I even created a Pipe that took any influencer ID and automatically created an RSS feed.

The idea was started because Daniel Tunkelang of Netflix was very anti RSS. I showed him, but alas, no more.

I never figured out how to use IFTTT. How can you grab any content from the web, like Netflix's json content? There is no HTTP input as far as I can tell.

I used a combination of Yahoo! Pipes and IFTTT personally. I'm not sure how people are using IFTTT on its own.

For me, I would do the data parsing on Yahoo! Pipes and prepare everything to be triggered, and then the actions would be performed on a service like IFTTT (where the instruction was to just post whatever new item came in the feed).

> "The idea was started because Daniel Tunkelang of Netflix was very anti RSS."

Um, what? I've never worked at Netflix, and I've never been anti-RSS. And I'm pretty sure there isn't another Daniel Tunkelang. :-)

I'm a big fan of Yahoo Pipes, and sad to hear it's being shut down. Pipes and Visual Basic shaped my view of how fun and easy programming can be.

We're trying to build on this legacy with Weld(.io). See a sneak peek of our programming UI at 0:40 here: http://youtu.be/faG3uuOnqxY

If anyone in the Pipes team would like to be involved somehow, ping me on tom@weld.io

I remember well using and playing with Pipes when it came out early 2007. The service worked. I mostly played around with my twitter and flicker feeds. Here's some screenshots of what it looked like.

cloning twit user: https://www.flickr.com/photos/bootload/385441178/

multiple feeds into one: https://www.flickr.com/photos/bootload/385433038/

extracting flickr data: https://www.flickr.com/photos/bootload/385101700/ and flickr backend https://www.flickr.com/photos/bootload/2789198106/

extracting twitter data: https://www.flickr.com/photos/bootload/385194782/

clogged pipes: https://www.flickr.com/photos/bootload/384133421/ and https://www.flickr.com/photos/bootload/384128224/

Read the comments on this link to get an idea of how I used Pipes and the logic behind it: https://www.flickr.com/photos/bootload/385101700

I have been a fan of Pipes for a while. Kudos to @psadri and others who made it possible with the state of web tech back in the days. It was a very well made tool both frontend and backend wise!

I have a emotional connect with Pipes as a tool. Right out of college, and before the whole API thing was pervasive, I had leveraged pipes for so many small projects and hacks. Infact I used it in a Yahoo Hackday hack and it landed me a gig at Y!

More relevant to the present, It has been something I have mentioned to so many people who have come to me discussing UIs for automation/control systems/or to manage workflows, processes. In a way it was what noflow's UI looks like, but years in advance. In a different parallel universe, it may be the way people use APIs, or right big Haddop/Spark/Storm jobs/topologies, and the front end was open-sourced back in the day, with full integration with hadoop for job management.

One of the truly powerful services, I will really miss. Mrs. Meyer squeezes out things at Y! like a lemon.

EDIT: added OSS & Commercial alternatives

free or commercial:

    [1] http://createfeed.fivefilters.org/
    [2] http://www.feedsapi.com/ 
    [3] https://theenginuity.com/search/
    [4] https://zapier.com/
    [5] http://www.elastic.io/
    [6] https://ifttt.com/

open source:

    [1] https://github.com/olviko/RssPercolator
    [2] https://github.com/fogbeam/Neddick/
    [3] https://github.com/cantino/huginn
    [4] https://github.com/fullscale/pypes
    [5] https://github.com/jparkrr/ISyTT
    [6] https://github.com/atask/shifttt
    [7] https://github.com/KLVN/F7_T7_RSSFeed

This a disappointment if not an unexpected one. I used pipes again just a few weeks ago to do some simple modification of feed but thankfully ended up taking another approach (not a pipes replacement I just wrote some code for it). Pipes was a really cool tool and it will be missed. Does anyone know of/use any alternatives?

As some deeper comment posted, Huginn [0] is awesome, an open-source connector-of-things like Zapier or IFTTT, but on your own server.

[0] https://github.com/cantino/huginn

In the enterprise space this concept is thriving and doing well. For example, SnapLogic has been building enterprise integration technology that can easily tie hundreds of different data sources together using pre-built or custom components using a GUI or programmatically. In order to become a critical part of business infrastructure it needs to be a professionally supported and maintained product. Yahoo Pipes was just an experiment and not a serious business so people never built anything serious on top of it that would pay for the underlying maintenance and infrastructure.

My website is partially powered by pipes. Looks like I'll have to rewrite.

I figured this was going to happen years ago. Any suggestions for a replacement?

Platforms like IFTTT Zapier have worked well for me, though they're certainly not direct 1:1 replacements.

Sure, check out http://www.feedsapi.com , it's probably the best replacement at this point, iit's however not a 1 to 1 replacement.

So sad to see Pipes go! I absolutely love Pipes and was just raving about it to a coworker this week. Terrific UI and a fantastic idea. When development fizzled out, I tried to find another service as a replacement, but everything I've tried has fallen short. I'm really disappointed as Pipes is definitely one of the best ideas that never caught on line it should. Popfly, while short lived, had the advantage of being able to modify the code behind, but it didn't come close to the fantastic ease of use and tremendous potential for reuse.

I always knew this day would eventually come. It's kind of sad.

I had created this years ago to display Twitter data using Pipes, YQL and Google Charts. I haven't had to touch it in years but it's always just quietly worked, and some people seem to like it:

http://xefer.com/twitter

; e.g.;

http://xefer.com/twitter/xefer

I liked that Pipes allowed services to be chained together driven from the browser.

Sorry to see Pipes being sunset. It had a major design and UX influence on a number of us during the early Web 2.0 days. Kudos to the Pipes folks for a seminal creation.

On a related note, a couple of us have been hacking on a Pipes like user experience around Data Integration and Analytics with Web feeds and APIs -- we had been planning a separate announcement but thought this might be a useful place to post if there are others who are looking for alternatives. Please message me directly - my contact info is in my profile.

If I was looking for a replacement, maybe I'd look for an RSS aggregation plugin for WordPress? Personally, I have my own PHP code that will slice and dice feeds however I want.

I got about halfway through building a "news blend" app where you tap just the categories you want and it would blend all the latest articles from just your categories into one news feed and email that to you daily. But we never finished it.

I'm really motived to create an opensource alternative as I was a big pipes user. I set up a repo (obviously empty for now) ,I'll try to come up with a minimal working app before yahoo pipes gets frozen:

github.com/interactiv/pipes

didn't decide yet what serverside tech i'll be using but quick deployement is a priority for me. If anyone's interested.

EDIT: a lot of useful information here about existing projects, thanks.

I loved yahoo pipes. I used it to filter high-traffic RSS to do a trivial word-match on content. I am now in the process of moving this to Newsblur, which allows filtering RSS content on tags, authors, or any word in the title. It's actually quite good, although we can't use the same filters for multiple feeds; at least it does the job.

I can understand why Pipes would be shut down by Yahoo, it was a impressive technical feat but it definitely appealed to very small developer focused crowd.

I feel like IFTTT is our best bet, while it doesn't do the "Unix pipe" that pipes offered it at least provides some usability to the process.

Using pipes was interesting and certainly did what it reported. I always capture what

Showing operators: https://www.flickr.com/photos/bootload/385194782/

back when pipes came on the scene it was so out there but it was amazing. We couldn't fathom what kind of magic was making it all work and there seemed to be so many possiblities. Future bright etc. Of course, landscape changed etc...but it was / remains a display of what is possible with the web and kind of the 'idea' of an open web/apis/all that jazz.

I am still using it to repurpose an XKCD feed into a fully formatted feed with images that then is picked up by IFTTT and tweeted out as a tweet with images when new comics go up. I used Pipes just because it was point'n'click and seemed to have all the key components I needed to do the task without worrying about the code-route.

Sad but not expected. It was really powerful but, like most people, I stopped using it years ago. It kept breaking either due to the sites I was using it with changing things or the service being neglected (and my limited ability to understand how it worked!)

Yahoo Pipes was probably one of the best examples of letting non-programmers be developers. I created a lot of useful pipes back in the day for friends and family. Used a particular craigslist one for years.

Disappointing news. But, if you're just looking at creating custom feeds for webpages, then try out Feedity - https://feedity.com

Pipes was certainly an inspiration behind what we've been working on at tray.io - sad to see this come to a close.

Hats off to all those that worked on the product - great to see it remembered so fondly.

Aw damn, that kinda sucks, but the project has been dead for a while now. I used it for content aggregation, but moved on to custom scripts, which were way more powerful...

As the one person still using Yahoo Pipes, I'm rather saddened by this. Does this also imply an eventual end to YQL I wonder?

Pipes gui was revolutionary when it came out and to me is invaluable. Shame to see it and rss die out from mass appeal :(

There was an awesome pipe which allowed you to get a rss feed of commentators you want to subscribe to. Sadly no more

I wonder how many people still use pipes?

One data point:

I used it for the first time 2 weeks ago.

Meetup.com has calendar feeds. You can get a calendar feed for all the events you've RSVPed "yes" to. You can get individual calendar feeds for each group's events. But there's no way to get a calendar feed of all the events happening in all of your groups.

I'm a member of something like 40 groups. The thought of manually adding each feed as a separate calendar in Thunderbird - and the thought of Thunderbird then trying to cope with the result - was... unpleasant.

So I set up a pipe to combine all the feeds into one. Very cursory googling suggested that Yahoo Pipes was the easiest way to accomplish this.

Fairly slick UI. Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.

Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.

That's really interesting. I think it exemplifies a problem that has been bothering me for a while: for many sites, scripting and design are antipatterns designed to corral rather than enable the user and leverage branding.

I used a pipe that created RSS feeds from Google+ pages in conjunction with some other stuff to get all the articles I'd want to read in one document (with a table of contents) delivered to my kindle. I'm now working on something that doesn't use Y! Pipes and simplifies the process.

We all knew it was coming, but the end of Pipes still sucks.

Sad to hear that. I guess Pipes was the first and probably the last innovative product ever shipped by Yahoo!

Pipes was awesome. But was there ever a credible strategy to monetize it? Does anyone on HN have any ideas?

First Google Reader and now Pipes? What's the problem with RSS and please tell me what's better?

please open source it, thank you!

in 2010 I attended Yahoo Hack Day in Bangalore/ India. Thats where I learned about Pipes. This was the only tool needed to create a tech mashup.

Thanks for your service! You've done great!

Web mashup! Hey, remember the late 2000s?

This is why I just roll my own data mining tools.

Reply on news.ycombinator.com