Building My Own Yahoo Pipes | Hacker News Reader

Building My Own Yahoo Pipes

taoofmac.com

116 pointsrcarmo15 comments

Sorry OT, but TIL about piku [0]

I have a old laptop in need of a new purpose. Might have a play with that.

edit: good post, otherwise!

Thanks! Piku has been my go-to for dev environments, Linux IoT and general-purpose stuff for a long while now, as it is simpler to iterate (no container build cycles, no need for a registry, relies on rock solid nginx/uwsgi, etc.).

I usually start with that and then containerize as needed, but this time it was actually the other way around -- since I didn't know what Node-RED modules I'd need and didn't see much point in going beyond a single core, I just sidestepped my custom Node-RED containers* and pushed out a Node-RED deployment (https://github.com/piku/deploy-node-red) to a underused, single-core piku instance.

(You can just steal the cloud-config from https://github.com/piku/deploy-on-azure and toss it onto any cloud provider)

(FWIW, I have a running k3s setup with the Azure equivalent of spot instances, but Node-RED can't take advantage of that setup and I mostly wanted the GUI and higher-level abstractions. You'd be surprised how much mileage you can get from something like this...)

* - https://github.com/insightfulsystems/node-red

This looks interesting. What is the service runtime controller in Piku? Is Piku mainly about configuring and controlling Nginx which directs Uwsgi that in turn runs app runtimes?

Also, does the k3s setup have publicly available provisioning code? Sometimes there's a need for temporary apps which happen to come with k8s configs.

The runtime controller is uwsgi, which acts as supervisor for all processes of an app _or_ embeds the right interpreter-it's pretty much perfect for that, as it also has a cron scheduler and an on-demand, fire-up-an-interpreter-upon-request mode.

In a nutshell, piku handles SSH git pushes, looks at the contents, figures out what language runtime to use, creates its equivalent of an isolated venv, and writes config files for every uwsgi process (which then start automatically and, if necessary, talk to nginx - which in turn understands what is going on and sets up cloudflare, SSL, etc.)

As to the k3s stuff, it's all here: https://github.com/rcarmo/azure-k3s-cluster. I update this from time to time as I occasionally nuke the entire cluster and rebuild it (I use AKS for non-hobby workloads, but a single master is just fine for most of my stuff).

Most excellent! We use Uwsgi in production with Django. I have often wondered why it isn’t more known. There’s support for many runtimes and features for larger scale. Only thing that’s hard is setting up configuration for your use case. Perhaps with Picu one could package best practices for various tech stacks. Then just serve the apps with ease. I’ll definitely take a look.

Thank you also for the k3s setup. Testing K8s apps locally and developing my own is far easier than in the cloud. Like making Picu work from within container and publishing apps under managed K8s. Without having to deal with kube things individually.

Yahoo Pipes were what the web was about: get data from any source, transform data the way you like, pipe it into whatever you want.

I only played with them a tiny bit, but was really sad to see them disappear.

Absolute full agreement. Repeating my comment from yesterday's forgotten submission¹:

Pipes was the only example of the "no/low code" idea that ever made me believe that possible future was remotely feasible. I saw a fair few co-workers produce some really interesting things with Pipes, and many wouldn't have known where to begin if you sat them down in front of a Python prompt with lxml. Seeing them chain filters, mangle date formats and input types was really cool.

The screenshots in the linked article make me hope that similar tooling can rise again.

¹ https://news.ycombinator.com/item?id=26529364

The nice thing about Pipes was (and still is!) that also if you know how to program the same task as script, having it in a flow based UI and using the pre-made blocks really makes these kind of web information retrieval tasks easier. On top of that not having to worry about how to run the pipe or script regularly on a server is nice, though the article here with the self-hostable solution had that hurdle again.

You can get Node-RED to run on PaaS services (I have other repos that deploy it to Azure App Service, for instance, which is "equivalent" to Beanstalk).

The thing about that (for me) is that Node-RED itself (and most of its modules) assume their execution context is a regular machine, so what it really likes is a normal filesystem.

But it does have a number of ways to schedule flows internally, so it's a pretty much all-in-one solution from the moment you get it going.

I saw that in your article a bit :) But Yahoo Pipes for me was about relatively effortless web infrastructure for me. A self-hosted solution would only be close enough to that if it was a one-click heroku image probably, and even then that would be problematic, since my account uses their free plan already for something else. That's a big part of why I went the SaaS route with my Pipes adaption and added self-hostability only later.

Great writeup, especially with all the workflow canvas screenshots. Makes me want to try out Node-RED.

From a few weeks ago on HN:

https://github.com/pipes-digital/pipes

“pipes.digital is a spiritual successor to Yahoo Pipes, a graphical interface to get data from the web and to manipulate it by connecting block”

These look like great tools.

I think calling these low-code / no-code frameworks is however quite misleading though, because there is actually code, it’s just inside the component! Code still has to be written.

One thing that these low-code / no-code frameworks do well is provide a structure that you can easily show visually.

This is particularity useful when you have to work with clients that aren’t that technical. It makes creating specifications much easier, which can be reviewed and revised before embarking on a big development effort.

I ran Huginn on my server for a long time until I finally found https://pipedream.com/ never looked back. I did try Node-RED at some point but as much as I liked the canvas, it just didn't work the way I wanted it to for my use-cases.

node-red is another thing I think of when I think about modern yahoo pipes

https://nodered.org/

That's what this article is about!

If they put the input/output on top/bottom respectively it would look a LOT cleaner than all of the repetitive snaking of pipes. Beyond that this is really cool and I miss the era of the internet pipes is from.

Reply on news.ycombinator.com