Show HN: patchbay.pub – Poor man's ngrok/IFTTT/serverless | Hacker News Reader

serverless

patchbay.pub

362 pointsapitman79 comments

Author here. I implemented[0] a stripped-down server that includes the MPMC functionality. Should be a good starting point for alternative implementations. It really shows how simple the concept is under the hood. All it really does is translate HTTP requests into golang channels.

[0] https://github.com/patchbay-pub/patchbay-simple-server

This is excellent - a big fan of its scriptability via curl. I think you have the beginnings of a truly amazing developer experience here.

If you're interested in monetizing this:

I would pay a fee for this - say $4/month. The only features I would like at this price:

1. Authentication (-H "Authorization: Bearer <token>" would be good enough) - at the level of the routes. This sounds like what you have done for routes like '/' anyway.

2. A CLI to tell which routes I "own".

3. Persistence of messages on the patchbay server up to some reasonable time limit T so that consumers can replay messages since time T without the publishers being blocked on their side.

I would pay $8 per month if:

1. You hosted consumers (go binaries or docker containers which would accept the body and headers of the HTTP response on the consumer side and do something with it), ran them for me in a loop

2. Guaranteed me some level of isolation for things like API keys, etc. which would be passed to my consumers in some fashion.

Even if it were open source, at these prices, I would buy. Don't want the headache of hosting it myself.

Thanks for the detailed suggestions. A few questions:

1/2. Why not a private instance for $4/month, accessible only to authenticated requests (unless of course you choose to open up certain routes for GET and/or POST)? Not only does this mean you can use semantic URLs instead of random ones (which helps offset the annoyance of managing API tokens), but you also get a unique domain that you can do things like put behind CloudFlare or another CDN.

3. This is an interesting idea. However, by far the greatest strength of the current implementation is that it has 0 (well ok, very little) explicit state. What's your use case for the semi-persistent producers?

1. It's interesting that you request hosted consumers and not hosted producers (a la for hosting a website). But I see the appeal. It would be nice to be able to set up something like a text notification trigger and not have to worry about it being tied to your own physical machine.

I totally agree about the open source bit. I don't see open source as a threat to a business model here. With something so simple it's going to come down to execution and customer service. As a staunch supporter of self-hosting, after 1 or 2 services it simply becomes too much to manage, unless you enjoy it as a hobby. I just want to get work done without giving my digital self over to the big guys. Even paying others per service doesn't really work, because at 5USD/service (which I paid for a Mastodon instance for a while), it quickly becomes too much money. Part of the inspiration for patchbay was trying to make a dead-simple substrate on top of which I could implement many other things, hopefully saving me some self-hosting trouble.

for 1/2 you could offer the route /<username>/<channel> to the paying user to help the namespacing issues and keep the shared infrastructure angle. Then charge "enterprise" pricing for private instances

That would go a certain distance, but I'd have to do a lot of measuring to see how scalable it is. Part of the problem with the free instance is I have to aggressively rate/bandwidth limit, because it's running on a single DigitalOcean instance with what maybe 1-10Gbps tops? I'd love to be able to give paying customers much higher performance by spreading across multiple machines. Though if any sort of a viable business model comes out of this, I'll probably look to switch to physical hardware anyway.

I mostly see myself using this as part of CI workflows or to monitor jobs that I run overnight/over the weekend. I also see it as a good way to, for example, start jobs on remote machines by sending a text message.

1/2. I can still make semantic URLs against your service, just with some caveats that they have to be globally unique. I could also proxy over to your domain from mine. Authentication would mean that some random person couldn't just come and POST or GET to my routes so I'm happy. $5/month for a DigitalOcean droplet on which I have to manage the server vs $4/month of using patchbay.pub where all I have to do us GET and POST HTTP requests, the latter wins every time (even though I can run other things on the droplet).

3. Semi-persistence would be great for failures on the consumers or on systems running the consumers. E.g. if I had a Comcast outage, I could still apprise myself of messages published during the outage.

(This last point is also why I would love for you to host my consumers - it also makes it a lot easier for you to make the consumers more reliable.)

Good point about paying for a large number of services - that's why I really like Twilio's pay-as-you-go pricing model. Might be a good one to consider for patchbay, as well - a per message published or consumed cost?

Anyway, really nice job. I think I'm going to use this a bit.

If you want to monitor my usage, I will set an "x-hn-user: zomglings" header on my requests. :)

> greatest strength of the current implementation is that it has 0 (well ok, very little) explicit state

I don't think that's the greatest strength. I think the greatest strength is how easy it is to get started using it and hooking it to personal tools. Whether it has or doesn't have an internal explicit state is totally irrelevant (to me anyway).

> 1. You hosted consumers (go binaries or docker containers which would accept the body and headers of the HTTP response on the consumer side and do something with it), ran them for me in a loop

I toyed with this idea at http://dollardeploys.com/ but the feedback I got was "Heroku can do this for free".

Noticed someone working on a rather interesting exploit in the logs:

POST /.well-known/acme-challenge/huhn2?mime=text%2Fplain

Plugged that one. If anyone thinks of similar holes I should consider, please shoot me an email at info@patchbay.pub. There are a whole new set of security issues that crop up with this sort of system. Definitely be aware of that before using it!

How did you "plug" that one? And what is "that one"?

I assume "that one" is posting to /.well-known/.

POSTing or GETing currently

I like this. Have you considered releasing the source so people can run it on their own servers?

^^^ This. The kind of people who use this are power users who want to run this on their server. Or use IFTTT otherwise.

Now that we've all seen the awesome implementation I think a clone in Go is not far off.

If I don't see one by EOD I'll have to make - is thing is super cool

You'd better post the code

Here's the extra poor man's version to get ya'll started. It doesn't have any of the auth/rate limiting/etc or pubsub, but it should basically work for the MPMC queue stuff:

https://github.com/patchbay-pub/patchbay-simple-server

Note that this will leak memory since there's no logic for removing old unused channels.

I did one too -- https://github.com/edoceo/httpubsub

Nice!

And, a bit late, me too, of the pub/sub option that sounded more appealing to me. https://github.com/olup/patchcreek

I would love to make the source available. But there's no getting that genie back in the bottle once you go that route. I wanted to release it and get some feedback before deciding what to do with it. That said, a server that supports the examples on this page is pretty trivial to implement (especially if you skip the pubsub stuff). I had a working prototype in a weekend. Multiple implementations would probably be a good thing.

EDIT: also, the current state of the code base is not something I'd be excited about having my name attached to ;)

Ah yea I totally understand.

I love this thing though. So simple but powerful.

Anders, please, opensauce it!

You can think about releasing the code with no license or with some form of the newer BSL or something.

It won't stop bad actors from stealing your code and using it but you will stop most corporate actors from touching it (and they are the ones you'd want to pay in the future anyway).

I made a similar project when starting to learn Go. It's opensource and it works. Just don't host it out on the internet. Intranet should be good because no security audit has happened. This was sort of an MVP

https://github.com/sairam/daata-portal/tree/master/daata-ser...

The name (including the .pub part) made me think this was related to the Secure Scuttlebutt client Patchbay: https://github.com/ssbc/patchbay

Same, I was super confused and I was trying to find out the relation the entire time and there is none.

No confusion was intended; sorry about that. The domains seem to be fairly orthogonal. Or am I not understanding what ssbc/patchbay does?

Patchbay is an interface for Secure Scuttlebutt.

Secure Scuttlebutt is a P2P pubsub system for sending files, messages (chat/notifications), and everything else you can implement with pubsub-like systems.

I think it's just confusing because of the overlap with:

- Patchbay

- Pub

- Channel

If you're curious about SSB, I've posted a link to your site on the network. You can view that (and other content) here: https://oasis-demo.fraction.io/thread/%25IGLa%2F3NDG3cj5gYOY...

Ah gotcha. I've heard about Scuttlebutt, but didn't realize there was so much terminology overlap.

There also used to be an IoT network called Pachube.

It was pronounced patchbay.

Well this is amazing, it's simple and functional, solves a wide variety of problems with a very simple approach.

Congratulations for shipping it!

Thank you!

I made a similar open source self hosted tool for sharing logs or analysis from several machines, hosting internal versioned documentation, counters and time-series graphs etc.,

This was from a few years ago - https://github.com/sairam/daata-portal/tree/master/daata-ser...

When people working in ops or with remote CLIs would have a great need for such a tool.

Added to https://github.com/fiatjaf/awesome-loginless.

Thanks! That's a sweet list.

Good luck with your project, it sounds interesting.

FYI, your domain is blocked by Cisco Umbrella (OpenDNS) as "This site is blocked due to a security threat that was discovered by the Cisco Umbrella security researchers." Not sure why, you might want to contact them.

Thanks for the heads up. That's fascinating. Has anyone else ran into this before?

Probably just a tainted IP from Digital Ocean.

Complete wild guess, but maybe it's because it matches an aggressive regex meant to match the pirate bay and derivatives.

Random cisco employee I annoedon IRC blocked my domain once, I complained to another cisco employee on another IRC and the block was promptly removed.

I don’t think they have much oversight over this stuff.

I ran into it this as well and the response from the folks running our block list said it triggered a "newly seen across all of OpenDNS" heuristic. This matches other similar blocks I've seen for newly registered gTLD domains.

Seems that this is the relevant documentation: https://support.umbrella.com/hc/en-us/articles/235911828-New...

How does this technically work? Is this a server which just tunnels through tcp traffic? So does all data pass through your server or is it a rendezvous server, which uses ICE to establish a P2P connection?

Yep everything goes through my server (see the security and privacy sections at the bottom of the page).

It would be awesome if someone could figure out a peer-to-peer way to accomplish the same thing. I'm not familiar with it, but I think that might essentially be what Tor hidden services are.

Thank you very much for your answer, and great project :) I am also not very familiar with P2P, but I know that it is a bit complicated because of NATS. ICE is a protocol to make it possible for clients to establish a P2P connection, but it is not always possible because of symmetric NATS. Thus, for example in WebRTC there exist STUN and TURN servers.

Ok, the first thing in my mind is how to share passwords with your teammates.

Sending a password:

   echo 'hunter2' | openssl bf - | curl -X POST --data-binary @- https://patchbay.pub/mysecretpasswordhandle

Receiving a password:

    curl --silent https://patchbay.pub/mysecretpasswordhandle | openssl bf -d -

This uses end-to-end encryption (Blowfish cipher) that relies on a shared password.

But how do you securely share the patchbay channel? ;)

invisible ink and paper

This is incredibly interesting and super clever. I love all the little examples of what you could do with this and am definitely going to play with this.

Thanks!

adoc formatted mirror from before/while this got destroyed from its chat getting DOS'd: https://pastebin.com/tAWNuRL7

Hm I'm not sure it was actually a load problem. The server load never spiked. I'm guessing the hosting got in a weird state, either on the server side or the producer process.

EDIT: But in case it happens again here's an internet archive link: https://web.archive.org/web/20191126170030/https://patchbay....

This is so cool! I really like your project. Its so simple and powerful and very succinct. Do you have an example for the "Poor man's file sharing"? I tried doing the simple thing:

> curl -F 'data=@test.txt' https://patchbay.pub/test.txt

And on the other computer:

> curl https://patchbay.pub/test.txt

But I'm seeing that the output is being wrapped with "Content-Disposition" and "Content-Type" at the top. I'm no HTTP expert, but is there a way to use your server (or namely, curl) to transfer the file so that it doesn't wrap this info?

Change the command on the sending computer to

    curl -X POST --data-binary "@test.txt" https://patchbay.pub/test.txt

The command on the other computer remains the same as what you had

    curl https://patchbay.pub/test.txt

Thanks!

Also want to mention that in general you can add mime=whatever on the consumer side to override the Content-Type the server uses. I haven't set anything up for Content-Disposition yet, so that's going to be whatever golang defaults to for the file extension. Feel free to open an issue if you'd like specific behavior

The term 'poor man' isn't very informative to me. Does that mean it has all the functionality but is free or does it mean has basic functionality and is free, or is cheaper?

I'm using it in the idiomatic English[0] sense, meaning a cheaper/inferior alternative. For any one of the examples, there are way better tools out there. But how many of those individual tools can be used to accomplish many different things?

[0] https://www.merriam-webster.com/dictionary/poor%20man%27s

I really like this. I think it'd be interesting to implement on my own server as well (You know, every non-cloud reliance or whatever). It's super simple and super easy to grok.

I'm guessing you aren't looking to put this out open source if you're looking to make a product of it, but I'd love to see the implementation.

This is great, I've been thinking of building something similar myself for exactly the use case you mentioned.

someone DDoS'ed the chat again, site is down :(

This is a fantastic idea. Great useful examples, the little chat client is such a novel use of a simple concept. Well done!

It would be nice to have read-only endpoints.

I thought about a couple different ways that could be implemented, without requiring a login. For example, you could make the channel a public key signature, then only accept POSTs that can prove they own the private key. The problem is that it makes it trivial for people to anonymously host whatever they want on my server...

As it stands now, there's a bit of game theory involved. You can shared a channel publicly, but if even one person doesn't like what you're publishing, they can bring it down. This essentially forces people to keep their channels for private use.

This is also one good use case for charging people for private namespaces/channels.

Simplest solution would be a different name for broadcasting channel. But you are right in your concerns about people being assholes.

Main problem with simply using a different name is that the server has to keep track of the mapping between the producer port and the consumer port. As it is currently the server has almost no state.

HN hug of death?

Not quite sure. The server load is surprisingly low, but it's gotten locked up a couple different ways.

Check the amount of traffic you've got: number of packets, number of connections. If it's under attack you might have LOTS of packets (for example SYN) coming your way which explains the slow performance.

Only a few hundred connections, and iftop is showing very low traffic (does that capture SYN floods?). I get the feeling people might be DOSing as a proof-of-concept then backing off, which I appreciate.

This is super weird. The "job queue" example made me facepalm, I'd probably use a one-liner like-

ls -1 *.mp3 | xargs -P4 -n1 -i ffmpeg -i "{}" "{}.ogg"

(yah the filenames would end up .mp3.ogg, but so does the example...should throw basename in there somewhere)

It makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).

You can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)

Sending the actual files would probably be really slow on the free server, due to bandwidth limiting.

As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash.

EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html

Yes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools.

# preserve filename for i in a b; do gzip -9 $i; done

# does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done

You might be better served with gnu parallel -

  #create the directory structure first
  find . -type f -name '*.flac' |ionice -c3 parallel -j1 'mkdir -p $transcodes{//}' &&
  #now the encodes
  find . -type f -name '*.flac' |ionice -c3 parallel -j4 "ffmpeg -n -v 0 -i {} -c:a libopus -b:a 128k $transcodes{.}.opus"

Note the {.} which would actually only grab the basename.

The ' should be " in first example. Otherwise $transcodes is not expanded.

Actually no, I'm handing that line to parallel which in turn will do the expansion when it invokes.

ls -1 *.mp3 | awk -F. '{OFS = "."; $NF=""; print $0}' | xargs -P4 -n1 -i ffmpeg "{}mp3" "{}ogg"

Reply on news.ycombinator.com