The Forgotten Yahoo Project That Inspired Two Recently Funded Startups | Hacker News Reader

The Forgotten Yahoo Project That Inspired Two Recently Funded Startups

nickoneill.com

159 pointsmikeknoop47 comments

YQL is an amazing piece of technology with robust infrastructure to support it. I was first introduced to it at a Yahoo Hackday in Bangalore and since then I’ve used it to to hack up a lot of small scripts. Like this http://userscripts.org/scripts/show/62104 - one that shows torrents links in Wikipedia movie pages(been couple of years since I updated it)

Christian Heilman was the evangelist behind YQL and it gained a lot of traction with him travelling all around to spread word - http://www.yuiblog.com/blog/2010/02/11/video-heilmann-yql/ . But YQL met with a quick death once Christian left Yahoo for Mozilla. Christian was also in the process of writing a book on YQL - http://icant.co.uk/whyyql/

The only major downside to YQL is that most robots.txt block the Yahoo Slurp & Pipes engine; the user agent that powers YQL. E.g. http://stackoverflow.com/robots.txt

YQL has a lot of hidden treasures. Matter of digging the community scripts to understand.

I continued to use YQL at hackathons in Zynga and stopped it when the primary author Nagesh Susarla left Yahoo - http://nagiworld.net/

Recently a Yahoo engineer told me that YQL is now used internally to power all services at Yahoo. So I am pretty sure it going very strong.

> Christian was also in the process of writing a book on YQL

Work on that has resumed and "YQL: The Definitive Guide" is due out next year.

> So I am pretty sure it going very strong.

Very much so. 100+ billion queries per month.

Pipes is maybe the only yahoo product I actually use. It's only for one frivolous thing though. Failblog (part of Cheezeburger I think? Doesn't really matter) Used to have a sub-blog called Hacked IRL. They shut it down a while back, but they still occasionally post clearly labelled Hacked IRL stories to the main Failblog. Since I liked Hacked IRL but really didn't want to have the whole fail blog feed subscribed, and I remembered having heard of Yahoo Pipes years before that, I looked it up and it was sure enough still around.

The result was this: http://pipes.yahoo.com/pipes/pipe.info?_id=5ead4957874f5a670...

Which is simply the fail blog feed filtered down to only posts containing "Hacked IRL" in the subject. I have the RSS of that subscribed.

Like I said, frivolous. But it only took a few minutes to put together even having never used pipes before.

Pipes was way ahead of the curve but is also targeted at a moderately more sophisticated audience than the task triggers IFTTT and Zapier. Pipes focus is more about data collection, filtering and transformation -- like Unix pipes vs cron.

BTW, both Pipes and YQL are powered by the same engine and serve billions of requests per day (for both O&O and 3rd parties).

I'd love to hear more about the engine that powers pipes and YQL. Can you tell us more or let me know where we can read more about it?

I'll encourage them to do a blog post on it.

The day Pipes launched was quite eventful. There was some very "creative" code in there that wasn't really designed to take the traffic it received on launch. Completely different today and I still use its cousin YQL for many things.

The Pipes/IFTTT concept screams for a more open solution, it's too important to be proprietary. I'm always frustrated when I want to use IFTTT but they don't support the services I need.

A simple protocol based on HTTP, JSON, and WebHooks could be pretty slick.

The protocol only gets you so far though. I'd argue that our (I work at IFTTT) value exists at a level above that. I would love a standard though, it would make my job waayyy easier.

It would be nice if someone would make a painfully simple interface (a la the "requests" python module) for doing XMPP.

Perhaps though XMPP is just too much of a quagmire.

https://developers.google.com/appengine/articles/using_xmpp ?

I've actually been working on exactly this (but considerably more powerful and in-depth than Pipes/IFTTT) for quite some time now (almost 2 years, on and off). It was originally designed to appeal to non-programmers; and I would have released it months ago but decided to rewrite nearly everything to make the code as modular/flexible/easy-to-learn as possible for developers (both newbie and seasoned).

Before I link to it, I'd like to apologize for overly generic descriptions currently on the site. I whipped together the entire site in a few days not knowing the exact direction I wanted to take the software, so I left everything pretty vague; I really just wanted some content that at least somewhat described my ideas. I'll put the link at the bottom of this comment so you read all this first.

From a technical standpoint, this is currently where I'm at with it:

- Users create a myriad of widgets that can "interact" with each other; the fullscreen version of Loggur consists of "layouts" of widgets, while the mobile version of Loggur lets you access widgets individually in typical mobile app fashion

- Databases are incredibly easy to create; just add fields to a widget and specify their relationships to one another

- All kinds of special extensions included by default, like automatic importing of various data sources, scraping of websites, cron jobs, PDF report generation, emails, sms notifications, triggers, and graphing

- Everything is taggable for reuse, from the apps themselves to widgets to components to elements to lists and to the data associated with all of the above; you can either "mirror" or "clone" any one of these parts in another app/widget/component by doing a quick search for tags (or if you know the exact path to the part, just enter that); so for example, if you really liked what someone else has made and wanted to reuse parts of it in your own app, you'd do a quick search for it, clone it, and modify it to suit your needs, saving a lot of time

- Data associated with apps can be any combination of public/private, singular (your individual profile), and/or group-specific; you can quickly/immediately switch between views of each

- Permissions on everything; specify who can view and/or edit apps, widgets, components, elements, lists, and/or data

- Appearances are somewhat customizeable and will become much more so at some point; customization currently consists of the basics like colors, backgrounds, and sizes; apps are designed to be scalable to any screen resolution (think large dashboards ;)

- Each one of the pieces outlined above (widgets, components, etc.) can be embedded on your own site(s) through small snippets of code

- Data associated with apps is easily accessible, currently only available in JSON but if for some reason other formats are requested in high numbers, I might do that

- Users can toggle the ability to view app/data updates as they happen in realtime; they can also invite each other (or a Loggur dev if they need help) to take turns using/building an app

- Regarding the mention above about rewriting the project to be more modular/flexible for developers, I felt doing this was 100% necessary/worth it because it occurred to me a few months ago that the best approach to make this succeed in the long term is to make this a legitimate platform (buzzword, sorry!) where developers can quickly/easily make and share awesome extensions and be rewarded (paid) for their work

Check it out (sign up for it ;) here and remember to ignore the bad, vague descriptions currently on the site: http://loggur.com

Sounds interesting. I'll give it a try asap you release it.

My memory is fuzzy about this but IIRC some folks from the Gnome Conduit project launched or talked about launching a web version of it. Not sure though.

Sadly, the project seems dormant. Could be an interesting starting point anyway.

The approach that I am working on is a web orchestration language. You can write s-expressions specifying the dataflow. Obviously alternative frontends can be built to output the s-expression.

I published more information on:

- Ideas: Egont, A Web Orchestration Language: http://blog.databigbang.com/ideas-egont-a-web-orchestration-...

- Egont Part II: http://blog.databigbang.com/egont-part-ii/

The problem with Yahoo Pipes was that it was over-optimized for complex dataflows. It was possible to do things like "find all Flickr photos from user, filter by tag and display on a map by location taken". The result was this crazy programming GUI that resembled Labview.

In practice, few people needed such complex querying and chaining. I think IFTT has a better metaphor ("if this, then that") which is much simpler but covers 80% of useful things.

I agree with you that IFTTT and Zapier are simpler solutions to more common needs, but man, Pipes is really powerful. It's more like a language than an automation framework. I don't know if it will ever make any money, but it's a hell of service.

Long before IFTTT, Pipes was an excellent solution for various social media needs.

I recall mashing up RSS feeds and using it to bypass various platform throttles.

It was awesome for fixing wonky feeds.

Love the UNIX metaphore and interface.

Pipes was the only Yahoo! product that really sang to me.

I think I've been relying on IFTTT too much lately. I may have to revisit Pipes.

... over-optimized for complex dataflows

Absolutely.

OSCMS 2007 was hosted at Yahoo!, and it was one of the first developer conferences I made it to after moving to the Bay area. My takeaway from the session on Pipes was that it was a bit ... advanced (at least it was for me at that time). Pipes was not necessarily "forgettable", but it wasn't quite to the place it would induce radical inspiration. The inspiration kinda got lost in the bog of over-engineered complex details.

This seems to be a common trend among Yahoo's many assets: build almost-prototypes of platforms that absolutely ooze potential energy. But then waste that energy by stretching it in too many directions at once, not getting very far.

> this crazy programming GUI that resembled Labview

Visual programming is a guarantee of failure, unless your target users are electrical engineers. Fact.

My first startup played in this space too, Grazr (http://techcrunch.com/2006/09/18/grazr-10-blasts-off-into-th...). We were way ahead of our time too (at least that's what I tell myself :) ). We were mostly known for an opml display widget but our intent was for the OPML to be a format for organizing packages of RSS (this allowed us to have lots of ready sources since feed readers all output OPML).

Instead of a graphical 'pipe building' interface, the way we did mashups was to have embedded scripts in the XML that ran in a server-side JavaScript environment (SpiderMonkey) extended with custom syntax and language extensions we called GrazrScript (and namespaced in the file as <grazr:script> tags). You could do some pretty amazing stuff with it by writing small amounts of server-side javascript. We had an interface for http querying, accessing and manipulating normalized feed representations.

You could have these files call other grazrscript files and create 'pipes' of functions (more directly like unix pipes).

I like to tell people we were doing embedded, server-side, javascript years before it was cool. :)

There are many good projects and services out there that scare/annoy away potential users by whacking a "register or login" box in their face just before things get interesting.

My identity isn't "free" and managing yet another account comes with a burden (another password to remember or store somewhere, another security risk).

Whoever thought that creating pipes (i.e. experimenting) without logging in was somehow detrimental to Yahoo's business interests(?), had things backwards.

Perhaps the login requirement was needed to prevent, or atleast reduce, abuse.

i've been a pipes user for many years. i mainly use it as a way to merge RSS feeds into a single bucket (e.g. craigslist items within 50 miles across multiple craigslists). my most frequently used pipe marries a bunch of news sources together by company name, a simple way to keep track of my industry and competitors.

i live in constant worry that yahoo will shut it down. glad to see it's getting some attention, and getting some clones, too.

Nobody seems to have mentioned - Kapow/OpenKapow - http://kapowsoftware.com - Dapper - http://www.readwriteweb.com/archives/dapper_quest_to_unlock_...

I remember giving up on YPipes back then because it was always down (or denied access). Did more with Dapper and OK

Yes, once I used Dapper more than Pipes http://open.dapper.net/

Does everybody trust into these hosted services? (I do not.) How about to run your own one? - http://pipes.deri.org - still alive or dead? - https://github.com/ggaughan/pipe2py - looks alive - http://neyric.github.com/webhookit/docs/index.html - Alpha and dead? - swiftly.org - started as a open source project, but nothing released and dead?

Did I forgot something/somebody?

Just to be complete, GUI as a JS lib: - http://jsplumb.org/jquery/demo.html - http://neyric.github.com/wireit

And additionally two commercial hosted projects: - http://www.wewiredweb.com (based on http://www.mashablelogic.com) - http://tarpipe.com/

I found pipes/YQL very useful for doing some ajaxy things with sites and services that are not on your domain (since most browsers don't allow for cross-domain ajax calls, you can request a JSON version of a site's markup via YQL). However I didn't find the service 100% reliable. It will be great to have more services like this out there.

I (and probably sriramk too) am disappointed the article didn't mention Microsoft Popfly: http://en.m.wikipedia.org/wiki/Microsoft_Popfly

Popfly was a more consumer centric mash up tool built by a little skunk works team inside of Microsoft's Developer Division.

Never heard of it. Because it is silverlight, never would have been able to run it. Sounds like it did a lot less according to http://blog.jonudell.net/2008/02/14/popfly-and-pipes/

Lemme guess, you worked on it?

Of course! No other reason to mention a Microsoft project that's been dead for four years otherwise :)

Hah. Well, you shipped something. I have to imagine that's incredibly hard.

The title made me think. Is there a place or archives where companies leave their forgotten or dead projects that they tried and moved on? Is there any HN's that dug up a failed project and turned it into a successful company/project?

I don't think I've ever heard of pipes to be able to have forgotten about it. However, after browsing the website for about 3 seconds I closed the window out of disgust with no idea what the product did. Compare this to IFTTT (which is fairly well known in this sphere) or zapier (which I had never heard of, but could figure it out after a few seconds on the website). It really might be all about presentation. Unfortunately I'm not willing to evaluate evaluate this statement in regard to pipes because I'm not willing to look at their website any longer to see if they actually do what the article claims.

There really is some really cool tech "locked up" in these big tech companies. Yahoo Pipes, Google Fusion Tables...

I think startups are, in a way, almost required to unlock the full potential of novel tech ideas. A startup can apply an awesome idea to a niche, gain traction, and expand whereas the defacto launch of a tech product at BigCo X or Y applies the tech across a broad segment of their user base. Broad launches are a good way to make your product generally available, but a bad way to ensure it truly solves some core need your customers have.

Yahoo pipes is great. And new automation platforms are fun and useful for non developers. But if you want to be completely freewhen programming the web, let APIs to developers.

http://api500.tumblr.com/post/28165124923/your-mashup-is-so-...

Pipes was, and I'd like to emphasis, still is a useful tool. A shame that Yahoo didn't market it better.

It's so easy to forget that every now and again Yahoo! does something actually really awesome like Pipes.

I remember looking at Yahoo Pipes back in 2007, but perhaps the timing was not right for it back then. Now as the Web is getting more and more ubiquitous and the need for services to talk to each other grows, an approach like this could very well take off.

Love the concept, but I found Pipes got very complex to do anything I wanted - a common problem with visual tools.

Zapier solves this by pre-configuring tasks that most people want. Also better times for it, with so many web-apps and web APIs around now.

Agreed. Pipes is cool, but as a developer, YQL is so much more intuitive. When I think of accessing data, I think of SQL & REST APIs, not diagrams. But, Pipes is good for non-developers who don't know SQL.

I came across pipes the other day while attempting to link my twitter feed to a (new) identi.ca account. The concept sounds powerful and I like the idea.. although I'd like to be able to view the source without logging in.

symbiotic of the problems at the rest of Yahoo. good technology, good ideas, good people, poorly marketed, poorly sold, poorly managed.

Pipe is extraordinary, but to complex and with a scary interface. Can't nave success.

I uses Yahoo Pipes for a project of mine—I depend on it, in fact. It's really useful if you're doing something service-based, say, if you need to make an RSS feed out of a website, or otherwise manipulate data. YQL (Yahoo Query Language) is more than a gimmick; it's useful.

It does seem to be under active development, and I hope it lives.

EDIT: here's an example Pipe that fetches a web page (Pinterest) and turns it into XPath, and then uses XPath queries to construct an RSS feed. Interesting stuff you can do!

EDIT: er, the URL is: http://pipes.yahoo.com/pipes/pipe.edit?_id=a08134746e30a6dd3...

I also depend on pipes. If pipes ever goes away, coding it all by hand would be a horrible pain...

Reply on news.ycombinator.com