Pipe Dreams: The life and times of Yahoo Pipes
retool.com
retool.com
This is the second installment[1] in what we hope will be a regular series telling the stories of novel programming environments that had a lot of influence on developers.
Creating these is a labor of love—we get to talk to the original teams who built products that were formational for us. We have an incredible in-house creative team that finds a way to sneak these fun mini-sites in amongst their other work.
For me, the most exciting part of working on a product like Retool is that it's an evolution of lot of ideas from the past. We're always looking at prior art to spark ideas and challenge our assumptions as we're building. There are many great ideas in old computing papers and products that were either before their time, or lost when there were major shifts in the industry. Pipes was a touchpoint for us as we built out one of our newer products[2].
We have lots of ideas for future installments in this series, but if you have any programming environments that were near and dear to you heart, let me know!
[1] Our first deep-dive was on Visual Basic. I'm very biased (I wrote it!), but if you like Glenn's Pipes piece, you'll probably enjoy this one too. :) https://retool.com/visual-basic
[2] Retool Workflows: https://retool.com/workflows
Generally speaking I don't think Linux is a target platform worth worrying too much about, but when it comes to programming-related content that might not be the case.
Maybe, but you’d think an iOS iPad might be a target. I just closed the tab, as there’s no reader view, and the janky scrolling and waiting for text to load (text!) wasn’t worth the effort.
Is there an RSS or JSON feed for just this series? If not, how about a webpage that lists them all?
I had a yahoo pipe that combined and deduped about 10-15 jira projects and about 30-50 GitHub commit streams into a single rss feed that I read through google reader.
It was glorious because it let me know quickly who changed what and when and where. And it was rss so I didn’t have to search through email or look at multiple notification pages.
i'm building a dev API on top of https://github.com/BrowserBox/BrowserBox that should provide the same functionality as a CORS-proxy / unrestricted iframe to be able to embed and interact with content across origins.
Like, smart people?
Not everything has a profit motive behind it. We have this entire concept called charities and philanthropy that also show cases how people will spend their own money to accomplish goals or viewpoints.
You've never hosted a fan site or a squad or guild site for a game or movie? Or ran a discussion forum? Or hosted an irc server for a community.
Not once did I expect payment for any of those.
It's pitched as being for Iot but it's incredibly flexible that you can do anything with it.
Alternatively, check out the source code at https://github.com/gorenje/cdn.flowhub.org and run it locally or somewhere else.
There are SaaS offerings for Node-RED or you can install it yourself but there isn't a one-click-Node-RED experience but https://deadred.flowhub.org provides that by being a serverless, static frontend-only version.
I couldn't get anything to work.
So much work to bounce your readers, pity!
I've never seen the phone lag on any other website. Hell, my Pixel 1 or Nexus phones didn't have issues either. I've never seen lag like this on any website, period... I don't think it's a GPU thing. Probably there is some condition that makes it really laggy on some devices and not others. Hmm.
It makes all Chrome windows unusable, when the tab with this page is displayed.
(Chrome on Android, Pixel 7)
Would definitely give it a try if you’re looking to automate Yahoo Pipes style.
I have no affiliation to them, just a happy user
No I don’t think so. You probably want n8n if you’re keen on self-hosting.
(disclaimer: I'm founder)
1 - https://github.com/turbot/flowpipe 2 - https://github.com/turbot/steampipe
I'll check out Flowpipe!
What's with this epic design and work put into this deep dive tho? What an effort from Retool. Suppose maybe just wondering why they didn't do it earlier! Ha
I can't speak for Retool, of course, but it was pitched to me by my fabulous editor there as the company wanting to show the great visual editing tools of the past and make sure they weren't forgotten. They had already run a wildly fascinating story about Visual Basic, something I never thought I would be interested in, and it was riveting. There was no marketing involvement in the story at all. There's an interesting line where producing compelling editorial work shines a nice light on what you're doing as a company? Stripe published Increment for years, both online and in a very lovely print edition, and I wrote (for the same editor) for years for them until that no longer served Stripe's particular purpose for it. But it was all essentially independent editorial with nothing to do with Stripe's core mission.
On the production side, I didn't see much of how it would look until it went live, and I think it tells an incredibly robust story—it's a great parallel to my reporting in a way that I've rarely seen in my whole career?!
It isn’t as simple, and there aren’t as many clickable components, but it was weird that I was so slow to realise it.
I guess someone with more insight can point out where I’m wrong.
[1] This Video Has 69,187,812 Views
https://www.youtube.com/watch?v=BxV14h0kFs0
A lot of what I thought were good, accessible, productive, entry to middle level technology ( HyperCards. Yahoo Pipes. Flash, Delphi etc ) never gained enough market. They were all the sort of middle ground good enough options. Instead technology is often filtered from the top end extreme. So everyone tries to use what Google, Facebook, Amazon, Microsoft are using. Because those tools are deemed as what Professional should use. The only rare exception that is powering most of the word is Microsoft Excel.
Pipes is dead, at least we still have ifttt.
A big example, I listen to a podcast where they have standard episodes and episodes where they play D&D and I just don’t care about the D&D ones so it was really nice to be able to filter them out.
So much was lost transitioning from web 2.0 to whatever walled dystopian garbage we have now.
The concept of pipes are way back from the Unix days, and the idea of this modularity and later building stuff from a flow based interface like this somehow never translated into anything on the frontend side.
I've been met with simple specs that had to be built with custom web solutions a few times now where i've almost started building a generalised model like this that could scale because most apps could be defined like this, as entities related with filters between them in sequences related to each other and web views where people could interact.
The closest i found back then was something like Airtable/Zapier combinations that was extremely limited.
Still miss Pipes though. It was awesome!
https://web.archive.org/web/20080312062739/http://developer....
Integrating IKEA five button remote with Sonos speaker.
Does the web need something like this today and in what form? All the replacements that are offered pale in comparison to excitement it was to use Pipes 15 years ago. Are we simply past that point?
The reason I think that interesting is because that's the model these days for everything from AI tokens to Monday.com seats. It makes me feel like Pipes was before its time.
That said I've been collecting different "business glue" products that are similar to Pipes. To me, like you say, they aren't as interesting, exciting and intuitive as Pipes was, but maybe it just takes a little more digging. I tried to focus on open source tools but some aren't.
- n8n io: https://n8n.io/integrations/mondaycom/
- Node-RED: https://nodered.org/ (just read about this one in this thread)
- trigger dev: trigger.dev
- automatisch.io: https://automatisch.io/docs/
- Activepieces: https://www.activepieces.com/docs/getting-started/introducti...
- Huginn: https://github.com/huginn/huginn
- budibase: https://budibase.com/
- windmill: https://www.windmill.dev/
- tooljet: https://www.tooljet.com/workflows
- Bracket: https://www.usebracket.com/pricing (just SalesForce <-> PostgreSQL)
- Zapier: zapier.com/
Anyway I hope some of these are fun!
Node-RED does great job in implementing the ideas of FBP in visual manner.
From each track in the RSS feed of tracks scrobbled on last.fm, do a google search and pipe the output back to an RSS feed into my reader.
Neither chatGPT nor Google could make sense of my request. Any ideas?
...something like that?
although this was also later deprecated in favor of Origami Studio, which retains the node-based UI: https://origami.design/
To call it a successor project is maybe arrogant and misleading - it can not do half of what the Yahoo project did, and is sorely missing time and work from my side to make it into what it should be. But if you are nostalgic for the concept and especially if you just want to filter or combine a few RSS feeds, or create one for a HTML page that is missing one, it's worth a try I still think. I personally use it regularly to get specific YouTube videos from a few channels that have many other videos I do not care about.
The article mentions one of the big problems for this kind of project: The APIs are missing or not open anymore. For example, a comment here mentions doing a google search for a song title and putting this in a feed - but you can't realistically parse Google. So you have to use the API, which I didn't know existed until looking it up now, and calculate the price tag attached to it very carefully. Maybe it would be a realistic integration to provide with a different search engine. Or a bigger project should run of the local search engines for the pipes block.
The article is also right in that there is a lot of influence of Pipes in different projects. I was always wondering why that influence was often so little - to react to output on a page in a wizard for example, instead of having blocks. I think I understand that a little bit better now - it is really a strange abstraction level. You are working with a flow of structured data, without necessarily having the correct data structure for that, and then that needs to be transformed somehow. The idea that you might have JSON on a page somewhere you want to extract and manipulate, but then that data is inside of your outer feed format - it feels wrong to confront the user with these details, it makes sure the target groups is developers, but it also does not seem like something that can be reasonably universally solved. But maybe I'm wrong here, and this is more a problem with my implementation (where RSS is piped from block to block) than with the concept itself.
The idea of a real Pipes 2 is very exciting to me though. Give it some initial funding, built all of the modern APIs into it that can be added (and in doubt just give users the option to use their own API keys) - especially things like ChatGPT, after all those LLMs are great in converting structured data - solve the data format question and crucially keep the interface block based (but maybe built with something like Flutter, so it can be better). Could work, not to win the world, but as something that can justify its existence next to ifttt and zapier. At least I regularly get the impression that the concept still fascinates people a lot, though their specific use case is very often very specialized and then often enough not something my small pipes project can cover. But if it were broader...
That is, IMHO, not a real problem at all. Definitely not what people make it to be.
Developers aren't alien species. They aren't a special class of people, bred and trained to commune with the machine. In general, anyone could in theory become one. For this particular case - anyone can easily become one: that is, through search and thinking figure out enough to use Pipes-like software to solve some need, improve their lives a bit.
Software like this is within the range of tips&tricks people exchange on-line, or from their friends. Idiosyncratic, Ad-hoc, low complexity, high-value. The random non-tech person who's e.g. carefully curating a Google Sheet of board games they play with friends, is driven and able enough to add some Pipes-like automation to it. They've picked up how to do conditional formatting and link images from a tutorial, they can pick up how to Pipes-in ratings out of BGG or something just as well.
Point being: total domination isn't the only measure of success. Will a Pipes-like product be usable for absolutely everyone? Not at all. But people with a need and minimum ability to use a computing device, or even a minimum willingness to learn operating a simple tool, is still a huge addressable market - easily an order of magnitude larger than "developers".
If anything, IFTTT and Zapier are proofs there is a market for this. The real problem is that everything these days is extremely locked down, everyone tries to control everything within their reach. As you note, there's hardly any useful API left anymore. Companies like Zapier succeed, because they sign contracts to integrate other services. They become a service aggregator, leaving user with no agency - whereas Pipes were all about users controlling their own automation. That part is hard to solve.
These days there's an abundance of public APIs that serve up various public information. What if you took a bunch of those, wrapped them up in a single typed SDK in TS or Python (using their existing SDKs if they exist), under a consistent devex, and just made it really easy to playfully string together different datasets
I actually was notified of this comment because I "follow" Glen on HN via this val: https://www.val.town/v/rodrigotello/hnFollow