Show HN: Pipe To Me – Stream data over HTTP using curl | Hacker News Reader

Show HN: Pipe To Me – Stream data over HTTP using curl

github.com

39 pointspyinthesky13 comments

Awesome. I would use this as shared clipboard. Immediately made bash functions for myself :)

pipefrom() { local key=${1:?"Error: Missing pipe key"}; curl -s https://pipeto.me/$key; }

pipeto() { local key=${1:?"Error: Missing pipe key"}; curl -T- -s https://pipeto.me/$key; }

Would there be no way for the server to indicate over HTTP to curl that it is not ready to receive until the receiver has connected? Then curl might be able to wait until it reads from the pipe.

Hello, Author here. I toyed around with the idea of not allowing the sender to connect until a receiver was connected, but I could see the use case where you wrote a stream to nowhere and then had receivers jump in and out.

I also toyed around with using the "-T." curl argument instead of "-T-" to allow some response in this case, but I came up against a known issue in curl (https://github.com/curl/curl/issues/932). I'm sure this wasn't the intended use case though.

It is definitely worth exploring further.

That is interesting. What made you stay away from returning an error if someone tries to post without a receiver?

I may end up re-thinking this, but one idea was that you could temporarily `tail -f` a log file into a pipe and then check up on it later via a mobile browser. It could be writing the whole time, but just dropping everything when you weren't watching it.

It would probably also make more sense if I were to add some kind of buffering or storage in the future.

I don’t know your needs and goals, and maybe it has to be HTTP, but what about handling the sockets yourself?

People could then use netcat to send and receive, and the server could block either end until someone connects on the other end. Personally I would really appreciate a tool I could connect to with netcat or anything else that speaks TCP. I don’t know how someone would use the service from javascript though.

No buffering needed by the server. If either end has to be asynchronous, just run in background + nohup if you’re calling this from a shell.

Maybe some inspiration from how blocking FIFOs work on Linux: http://man7.org/linux/man-pages/man7/fifo.7.html

You could even do a non-blocking version which just refuses connection. netcat can be set to retry: https://unix.stackexchange.com/questions/432249/nc-to-retry-...

I realise that this doesn’t make it easy to indicate which pipe one wants to connect to.

I think you are right actually. I will experiment with some ways to make blocking vs non-blocking operations. I need to try out @paulddraper's suggestion below to see if I can do that over http. I'd like to make a raw socket version regardless. Like you said, I just need to figure out the best way to pass the key.

Edit: One nice thing about the https interface is that you get an encrypted channel. Maybe SSH would be another option for transport. That would also make passing keys easier.

How would the SSH work?

tl;dr Yes an HTTP mechanism exists, if you use curl --expect100-timeout=<large number>

The HTTP request header Expect: 100-Continue and HTTP response status 100 Continue informs the client that the server desires to receive the request body.

However, many servers don't support it, and curl proceeds anyway if there is no Continue response after 1000ms. This timeout was originally fixed [1], but since curl 7.47.0 can be adjusted via --expect100-timeout.

[1] https://curl.haxx.se/mail/lib-2010-01/0182.html

What are some practical applications for Pipe to Me? (should be in README.md)

There are a couple (admittedly simplistic) examples in the readme like transferring or remotely watching files. The basic idea though is to take data from a unix/linux pipeline from one machine to another with the benefit of also being able to watch it in real time in a browser.

how does it update without javascript?

The browser will start displaying the contents of a response before it has entirely downloaded. So in this case the browser sees it as a really slow server (you can see the loading icon is still there in the demo) that sends a bit of content every so often

Reply on news.ycombinator.com