Show HN: Station307 – Stream files between cURL, Wget and/or browser | Hacker News Reader

Show HN: Station307 – Stream files between cURL, Wget and/or browser

station307.com

45 pointshakvroot16 comments

Author here! This is a little side project I created because I found myself using scp + email to send a file from a server to someone sitting right next to me more than once. I have a different project which does relaying over HTTP so I figured it might be a good idea to reuse a part of the codebase for a simple file streaming service. I finished it yesterday and it already proved useful for myself, and I hope it also proves to be useful for you.

If there are any (more) questions on the what or how, I'm happy to answer them. The day here is however coming to an end so it might take me a couple of hours. While I'm gone, please be aware that the service is running on a $5/month VM ;).

Hey really nice to hear from you. Would be really nice if you share source code with the community. I really like the HTTP status code 100 trick :D

Thanks! I made some comments regarding to open sourcing it in reply to vortico (this thread) - current status is contemplating.

Very nice. I've bookmarked it.

By the way - there was another similar service recently, but stores files: https://transfer.sh/ , https://news.ycombinator.com/item?id=11322007

Thank you! And thanks for linking to the thread as well; I did already come across it but it definitely contains some good discussion on general feasibility, DCMA issues and other similar projects. It was also nice to learn that transfer.sh is also made in Holland :).

I've recently wanted to write a streaming download service like this recently. The great thing about your service is that although it's a proof of concept of a handful of neat tricks, the core functionality works perfectly and could be extremely useful. This makes it faster than ever to drop upload links in a chat room and effectively solves the problem in https://xkcd.com/949/.

Is the backend licensed as free software? It would be fantastic for admins to host their own version of this since it allows the concept to propagate without worry of shutting down.

Currently I haven't licensed the backend as free software; the primary reason for this is that I reuse quite a lot of components from a different project of mine (https://gemin.ee) and I haven't decided yet on whether I want to release those components or remove them as dependencies. Since I too am familiar with useful stuff suddenly disappearing I'll give you my word that I will release the source if I have to shut down. If I decide to share the source I'll post a link to it on the site itself, so you won't have to look for it ;).

On another note, I'd encourage you to write a streaming download service. Compared to file uploading - storing - downloading I found it fun to do but also quite non-trivial to accomplish. I am planning to have another go at it myself, but with a different language (Go, Pony or Elixir). When run alongside of each other with the same UI/frontend/api it would make a nice benchmarking mechanism :).

How does this work? I see the pending POST in the Chrome network tab, which completes when someone downloads it, and then become pending again for the next download. So are you proxying the uploaded data to the downloading client?

That's exactly what's going on. In a bit more detail: when a POST arrives and no receiver is available yet the server lets the sender know that it won't handle any more data (TCP Zero Window). When the receiver connects to the server the server asks the sender to continue writing and the data is relayed (proxied) from sender to receiver. If multiple receivers connect they're put into a queue and the server waits for the sender to POST the data again once the ongoing transfer is finished.

An interesting difference is that in the browser the rePOSTing is done through scripting whereas when using cURL or Wget (or httpie, or ...) the server will issue a 307 Temporary Redirect allowing cURL/Wget to send the file again without manual intervention. Hence the name! (although that doesn't really account for the "Station" part yet)

Hi! Thanks a lot for sharing details about it, it's an ingenious app. :)

Could you please elaborate a bit on how you work with the TCP window size -- how might I set it to 0 and back? Do you simply use SO_RCVBUF or is there more to it?

If you happen to go through with making more variants of this, it would be amazing if you could make them open source as well (even just the backend). It's a clever idea that solves a very real problem!

Thank you! And I'd be happy to elaborate a bit more, although you already explained most of it yourself ;). Currently the server just stops reading from the socket and the rest is being handled by the networking stack (i.e. buffer fills up, window size is to 0). At this moment the buffer size is also fixed (it sits somewhere between having good throughput and not consuming too much memory for parked connections), but an improvement would be to change it dynamically to improve behavior on both fronts.

For any other versions or source releases I'll post an update!

This is very useful! How do they think to earn money with it?

Thanks! I don't have any plans to monetize this to be honest. I built this because I had the need for this myself and, if people like it, it will be a nice exercise in maintaining something that works well under stress :).

Looks great! What is the technology stack behind this, if I may ask?

Thank you, and of course you may.

The backend is written in Java(8) and it is built with the following (noteworthy) components :

  Netty[1] - Extremely fast and stable Java networking framework,
             serves static files and relays traffic 
             (both tasks are custom implementations)

  Guice[2] - IoC container by Google, nice for keeping the code clean

  jOOQ[3] - I log some statistics in PostgreSQL 
            (file size, content type, transfer time - but no filenames). 
            If you're comfortable with SQL jOOQ is a real breeze to use.

The backend is hosted on the cheapest VM option at Digital Ocean (Amsterdam region), (which seems to be holding up pretty well so far :). I do run PostgreSQL on another $5 VM, so the Station307 server only has to worry about serving static content and moving large chunks of bytes around. SSL certificates are kindly provided by Let's Encrypt[4].

For the frontend, I know for a fact it uses left-pad (babel-core)... Written in JavaScript (ES2015 + JSX) and built with :

  Mithril[5] - Fast, small and simple React-like framework, quite a joy to use.

  Bootstrap[6] - Because not having to worry about L&F is quite nice. 
                 Trimmed it down a bit by only importing the parts I needed (SCSS).

And I think that's about it. If there's anything else you'd like to know, let me know!

  [1] http://netty.io
  [2] https://github.com/google/guice
  [3] http://www.jooq.org
  [4] https://letsencrypt.org
  [5] http://mithril.js.org
  [6] https://getbootstrap.com

edit: formatting D:

WOW this is so cool

Reply on news.ycombinator.com