Ffsend – Share Files from the CLI: A Firefox Send Client Written in Rust | Hacker News Reader

Ffsend – Share Files from the CLI: A Firefox Send Client Written in Rust

gitlab.com

306 pointshugodutka102 comments

Another really neat solution to sending data between two machines that don't have a pre-existing channel is magic-wormhole[1]. It's much simpler because it's just a number and some easy-to-distinquish code words like 1-stethoscope-zulu.

[1]: https://github.com/warner/magic-wormhole

Very cool!

I dove in to see how they solve the routing / firewall problem. Turns out you’re relying on the kindness of strangers and a public WebSocket relay. Hard to see how they could do any better, alas.

IMO one of the big holes in the architecture of the web is the lack of a standard peer-to-peer “content-addressable routing” protocol, perhaps based on any of a number of distributed hash table schemes. It’d be perfect for something like this.

Far from a standard but have you checked out libp2p? It's pretty far along as a modular p2p networking util.

https://libp2p.io/

Excellent! Yeah, I often daydream about a web where something very much like that had become foundational.

Unfortunately, the academic work on DHTs emerged at around the same time as both the stultifying dominance of Internet Explorer and the panic over Napster et al.

A better solution for p2p E2EE file-transfer is croc [0] because it can compile to a statically-linked binary without runtime dependencies.

Sending files: `croc [files]` Receiving files: `croc code-phrase`

Custom code-phrases are supported.

More information on design decisions are documented in a blog post [1].

[0]: https://github.com/schollz/croc#usage

[1]: https://schollz.com/blog/croc6/

> A better solution for p2p E2EE file-transfer is croc [0] because it can compile to a statically-linked binary without runtime dependencies.

For some definition of "better", sure. I really do wonder why the author didn't make it compatible with magic-wormhole. Seems a bit silly to cause incompatibility between two implementations that are so similar that they are functionally identical.

> Custom code-phrases are supported.

magic-wormhole supports that too (you still need to have a channel number but you can pick that too).

The one feature of croc which magic-wormhole doesn't have is transfer resumption (though I do wonder how difficult it would've been to add this to magic-wormhole -- or to at least extend the magic-wormhole API to support it). As far as I can tell, the only other real benefit of croc is "it's written in Go" -- which is a real benefit for some people (most notably, Windows users without a working Python setup) but it's not really one that justifies having an incompatible version of basically the same idea.

Just a note, there is a Go port of magic-wormhole that does properly interoperate with the official python version and has the benefit of not requiring a python environment: https://github.com/psanford/wormhole-william

Oh neat, that's quite cool. I was hoping to write an Android app for magic-wormhole -- so you could share files with mobile devices. Packaging Go on Android is much nicer than Python, so I might revive that idea at some point.

I have a very (very) rough react-native frontend for this that runs on android. I've been waiting for the go-mobile module support to settle down a bit before cleaning it up and releasing it. Its probably about time for me to pick it back up again.

This is not to discourage others from building on top of wormhole-william. I built the public API[1] first and then added the CLI tool with the hope that others would be able to easily pick it up and use it in other applications.

[1]: https://pkg.go.dev/github.com/psanford/wormhole-william/worm...

Developer here. Thanks for posting! Feel free to ask my any questions.

Upload is very slow for me, about 40 kB/s compared to the about 400 kB/s I get with scp. Is that normal?

Edit: It seems equally slow to me when using the website, so I suspect the issue is with Mozilla's sever, not the client.

If might be because of slow cryptography, that isn't hardware accelerated. What system are you running on? I can easily reach 20MB/s.

Feels too slow even for slow cyrpto. I suspect either that I am rate limited by the server or there is some bug somewhere.

CPU: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz

Kernel: Linux 4.19.0-6-amd64

Distro: Debian 10.2

Rust: rustc 1.40.0 (73528e339 2019-12-16)

Program: ffsend 0.2.57

That should definitely work fine, yeah! I'm assuming it has to do with the network or their server as well. Sorry I can't be of much help.

I mostly use Firefox Send to transmit data between my own devices and devices I do not trust. I wish the web client could generate QR codes to make that easier.

Probably I could combine Fsend and another CLI tool to enable that use case. Right now I'm manually typing links into different devices.

EDIT: Fsend can do this by itself!

Just want to note that QR code generation is a feature in `ffsend`. Simply use the `-Q` flag while uploading.

Wow, that's great! Thanks.

Printing an ascii QR code to stdout would be a pretty cool feature.

According to the author, use -Q.

That's pretty neat. Currently I use dat, resync or ipfs to move big files around on CLI. Maybe I will try this next time I wanna share big files.

Obligatory web alternatives that are peer to peer -

[1]: https://github.com/kern/file.pizza

https://file.pizza

[2]: https://github.com/cowbell/sharedrop

https://sharedrop.io

[3]: https://github.com/webtorrent/instant.io

https://instant.io

[4]: https://github.com/RobinLinus/snapdrop

https://snapdrop.net

I like that this removes some of the need to trust the service. When you open a link in a browser it's tedious to verify that the webpage doesn't send the secret to anyone, but this way you only have to trust the command line tool, which can't be replaced unpredictably.

Somebody that knows zilch about Rust with a question: I like to write CLI tools with Go, the built-in libs and the simpleness of the language make it really easy for writing good tools quickly. What advantages does Rust have over Go that one would use it for this purpose?

I'm not certain, just learning it. You might find this blog post that does a code review of ripgrep interesting, though. It's a deep dive into a serious-scale cli tool.

https://web.archive.org/web/20190401042439/http://blog.mbrt....

Here's what the main function looks like:

    fn main() {
        match Args::parse().and_then(run) {
            Ok(count) if count == 0 => process::exit(1),
            Ok(_) => process::exit(0),
            Err(err) => {
                eprintln!("{}", err);
                process::exit(1);
            }
        }
    }

I don't know the current state of go package management but I do know the equivalent Go project cited Rust's package manager Cargo as a good reference for developing their own. You forget Rust's standard library is quite minimalist and you get to sample competing libraries which often use interchangeable interfaces. In particular there are several crates (libraries) which make handling CLI arguments convenient.

If your code is stateful the typestate pattern is particularly easy to write in Rust - http://cliffle.com/blog/rust-typestate/

I've never heard of the TypeState Pattern, thanks for the link and it even got me interested in getting to know more about Rust. Thanks a bunch!

It's 2020. Is it finally possible to send a file over the Internet directly from one machine to another?

Only if you are not behind symmetric NAT I suppose.

Tangent: why does symmetric NAT exist? It seems like the kind of thing firewall fetishists do "for security" without considering the problems it causes and without any clue of what sort of attack it thwarts that mere stateful firewalling does not thwart.

Realize that without symmetric NAT, TURN would be largely unnecessary.

I am pretty sure there are many people on here who have not forgotten (it is not even that old! :P) and still use any variations of netcat, or even rsync. Nmap's Ncat is cross-platform (I use MSYS2 on Windows so it does not matter much to me), supports SSL, proxies, connection brokering, and a lot of other stuff[1][2].

[1] https://nmap.org/ncat/guide/

[2] https://en.wikipedia.org/wiki/Netcat#ncat

All that stuff doesn't work between systems behind firewalls or NATs.

> A basic file transfer is very simple: Start Ncat in listen mode on one end, start Ncat in connect mode on the other end, and pipe the file over the connection. There are two ways to do this that differ only in which end listens, the sender or the receiver. Sometimes you can't create a listening socket on one end of the transfer because of a lack or permissions, NAT, or filtering. As long as you can listen on at least one end, though, you can use this technique.[1]

This works with most if not all netcat implementations.

> The basic file transmission technique described at the beginning of this section fails if neither participating host is capable of listening, or the two hosts can't communicate directly. This situation has become common with the prevalence of network address translation. A way to work around it is to use a third host as an intermediary. The intermediate host listens in connection brokering mode and the other two hosts connect to it. Recall from the section called “Connection Brokering” that in connection brokering mode any input received on one socket is copied and sent out to all other sockets. With just two hosts connected this is especially simple: anything coming from one host gets forwarded to the other. This example shows host1 sending inputfile to outputfile on host2, using host3 as an intermediary.[1]

This works nicely with Nmap's Ncat.

[1] https://nmap.org/ncat/guide/ncat-file-transfer.html

The second is not direct.

magic-wormhole solves this problem (while being encrypted and incredibly easy to use). If the two hosts can directly reach each other, it will directly do the transfer, otherwise it uses a public relay server which both hosts can reach.

You could use https://github.com/rtctunnel/rtctunnel to setup a proxy over webrtc.

I'll go ahead and plug https://patchbay.pub here. If you're somewhat less concerned about having encryption, you can do CLI transfers of files of arbitrary size without installing anything (assuming you have an HTTP client like curl already), and without needing an account:

Sender:

curl https://patchbay.pub/random-channel-chosen-by-you --data-binary @file.mp4

Receiver:

curl https://patchbay.pub/random-channel-chosen-by-you

As mentioned by others, https://file.pizza is great if both devices have browsers, and the WebRTC connection between them succeeds.

Your service sends everything through the instance so we rely on the bandwidth you're willing to share. It already is amazing, but the same thing through WebRTC would be really great. I know it means work would mostly be on the client side though, and curl wouldn't be enough anymore

These are all very interesting, I am currently facing an issue getting BitTorrent file transfer to pierce an "unfriendly" firewall that I can't change permissions on. I've tried a VPN out with no luck. I wish I knew if one of these would allow me to get out.

Firefox Send is a web page, so if you can access the web you can use it. Also, a firewall that you can access the web from will have ports 80 (http) and 443 (https) open, so configure your tools to use one of those (preferably 443). If both machines are standard internet-facing Unix-likes that you have accounts on, you'll already have scp preinstalled for easy file transfer.

Many modern firewalls do Deep Packet Inspection, so obfuscation might be needed too.

As long as your traffic is encrypted (which ssh/scp is) and as long as your machine isn't compromised then a firewall can tell so little about your traffic that it would have to be pretty paranoid to attempt to block any of it. It'll work at a Starbucks, but perhaps not in China.

Deep packet inspection will almost invariably be able to recognize ssh/scp from the packet sizes, the patterns, the ports and so on. You really do need some kind of obfuscation. Starbucks won't catch it, but I know for a fact my former high school does, for example.

Shadowsocks can get past most. Many have yet to update to catch wireguard, too.

this would work for me if I either had alot of patience or a different dataset. I can either send 1 70GB file or 70 1GB files.

few more similar (python based) command line options for Firefox Send - https://github.com/nneonneo/ffsend & https://github.com/ehuggett/send-cli

Would be great to have something like curl -X POST @myFilePath "https://api.send.firefox.com"

Looks cool! Personally, however, I’ve been using croc (https://github.com/schollz/croc) and can’t recommend it enough. I use it almost on a daily basis and is a super easy file transfer tool.

It's crazy how incredibly similar to magic-wormhole[1] this looks.

EDIT: Ah, they mention magic-wormhole in the acknowledgements as being an inspiration.

[1]: https://github.com/warner/magic-wormhole

Yeah, I made croc because wormhole wasn't easy to install on Windows and wormhole also doesn't support resuming transfers.

Great tip. It's astonishing how well done the packaging is; all platforms are covered.

Except mobile; it would be quite awesome if that were to exist as well.

termux + woof

woof -i <ip_address> -p <port> <filename>

termux: https://play.google.com/store/apps/details?id=com.termux

woof: http://www.home.unix-ag.org/simon/woof.html

1. Allows directory upload/download (tar/gzip/bzip2 compressed)

2. Local file server (doesn't need to go over the internet)

3. Allows upload form (-U option)

4. Allows file to be served <count> number of times (-c option)

What does the fact that it's written in Rust do? Do we get more storage space? Can files being downloaded faster?

It might use less battery time?

Thanks for your extremely useful sarcastic comment. Maybe the author just wanted to play with the language and share their knowledge through a cool open-source tool. Maybe it's just the best language for the job.

yes but they do have a point. 'Written in Rust' adds nothing to the tool itself.

Edit: I see now. Thanks

That's not true! See other comments in this thread.

It does actually a lot - I know that when I contribute to this project I have a much lower probability of shooting myself in the foot compared to most of the other languages

Aren't many of us programmers? Don't we gain something from reading source code in languages we might be interested in using?

We are craftspeople. Of course we care about how tools are made. Else we'd be at ProductHunt.

Conveys the ecosystem. When I see “CLI tool written in Python” vs “CLI tool written in Rust/Go” I pick the latter because Python’s packaging situation for command line tools is beyond awful.

By packaging situation do you mean the user-side of things, i.e. how to install stuff separately of each other? For that I've had good experience using https://pypi.org/project/pipx to install several cli tools without conflicts. Uninstalling them has also been a breeze.

I wrote a tool called pipsi which I believe has influenced pipx but despite the existence of such tools I have no idea these days how to install python packages any more. Ir has become only more confusing and complex. Both for cli tools as well as libraries.

Pipx is the replacement for pipsi. It's good for installing cli tools.

For managing project dependencies, Poetry is the current best. It means I dont have to interact with virtualenv or pip or setuptools or pip-tools or manifest.in.

This is very surprising to read. When I read something like this from others, I was always arguing that "it's not anymore", but I'm absolutely sure you know better than me. Would you mind elaborating actual problems or writing a blog post about it? I am VERY curious about your experience and opinion in this topic!

Personally it's more about familiarity, I dev Rust so I know that I have all the prerequisites and setting this up should be trivial, meanwhile I don't do a lot of Python work so getting that to work might be a bigger commitment.

I don't do a lot of Go either but at least Go generates freestanding binaries so if I install it in precompiled form I know that it will be fairly frictionless and won't rely on half a billion dependencies (which may or may not conflict with something else). I use a bunch of Go programs regularly (like docker or Arch's yay package manager) and it never gets on in the way.

On the other hand if it's an npm package I don't even think about it.

To me, it sets expectations for

- single binary deploy(unlike Python)

- fast (llvm)

- memory efficient (no GC)

- newer (by virtue of the newness of the language itself).

Nothing wrong with setting expectations in title ;-)

GCs have come a long way. I think the bit about being memory efficient for not using a GC is FUD.

>> GCs have come a long way

Aha. I have a web server written in rust that literally uses about 5Mb memory on RPi. And that one actually has a proper web framework, not something barebones.

Good luck doing that in Python.

I was thinking about Go not Python when writing the above.

Which crate/framework do you use ?

I'm looking at REST API frameworks for rust these days, but nothing looks decent...

I went with Rocket. Basically I wanted to control the switching of devices on my sound amplifier using USB infrared remote. So I built the UI using yew and the web server using Rocket - but I wanted to do it fast and only in Rust.

But I bet it took a lot more code than:

  python3 -m http.server

Note also that Python is primarily reference counted, with GC used only to detect cyclic references.

I am not serving static content, so this reply is absolutely irrelevant.

On top of this, you can do the same with docker run nginx -v.:/var/www (which btw is a similar number of symbols). Also python’s http module barely supports anything but the most basic content-types (so good luck with eg SVGs).

At a few more orders of magnitude of memory use, sure.

Tradeoffs everywhere, which make sense or not depending on your requirements.

But Python's ref counting isn't a significant factor in its memory use.

>> At a few more orders of magnitude of memory use, sure.

Are you implying nginx uses more memory than a python interpreter? :)

Reference counting is a form of garbage collection.

Sure, if you want to be pedantic, but the context was limited memory use, and ref counting does not increase the memory used for unreferenced objects (except in cycles) unlike most other garbage collection algorithms.

Even just loading the base Python interpreter uses more memory than heavenlyblue's entire server.

  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root     23612  1.2  1.7  62412 17492 pts/1    S+   12:38   0:00  |       \_ python3 -m http.server

I suppose it depends on how low you need to go, I have a service written in Java EE that is using 13Mb of memory.

I lied. My server actually uses 2.704mb memory.

On top of that, that server doesn't actually use any sorts of downgraded language version that Java EE is.

You literally get that for free with Rust.

I still don't understand why people argue against this. Rust is a better language in so many aspects (tooling, language capabilities) it's really hard to argue against it.

In practice it is quite common.

No GC forces the programmer to think about memory, this is a good thing because you need to to that in all garbabe collected languages anyway - or you will get bloat.

For most use cases I consider garbage collection to be a mistake. Scripting languages or small one-off applications exempted.

GCs have a nonzero overhead wrt manual management. This is an undeniable fact. The only time they can compare is for memory allocation profiles that already "emulate" a sort of a tracing/refcounted GC themselves.

Many GC enabled languages offer other ways to manage resources as well.

You can have single binary deployment with Python.

Most people don't, though. They provide some install instructions using pip in their README, and IME these barely ever work without modification.

Yes, but every time I tried it, it always fails me with some obscure error that I run away from it with bitter memory(why did I do this in Python!?!). If it is not a priority for the language, I’ll rather choose a language which cares about it.

How? Are there reusable tools for this now?

I've used PyInstaller at least. Pretty sure there are others as well.

I found the success rate of this very low and it’s very hard to automate this for all platforms.

I've never really used Python, but as a consumer of tools, I don't recall ever coming across a Python tool packaged as a binary - so it might be possible, but dose anyone actually do it?

youtube-dl does at least.

Unnecessary sarcasm aside, I'm Keen to look at it simple because it's written in Rust. I like Rust so this is interesting for that reason alone.

Besides that, Mozilla and the Firefox community have a lot of Rust devs so writing this in Rust may make this more likely to get PRs from people involved in the FF Send project

Nothing. You can get the same secure sharing service with zero hassle with a simple shell script:

    sum=$(sha1sum "$1" | cut -f 1 -d ' ')
    ssh your-vps mkdir /srv/http/files/$sum
    scp "$1" your-vps:/srv/http/files/$sum/
    echo https://your-vps/files/$sum/$1

+ nginx or whatever

That isn't really comparable in that it's not E2EE

It is, as all ends belong either to you (source computer, vps) or the destination device, and it uses ssh and https.

The VPS doesn't really belong to you. Google has exactly the same ability to view your secret file shared unencrypted over Drive as they do to view your file stored on your f1-micro instance.

That's just your assumption. VPS != Google. It's just a virtual machine running on some HW. I agree that if that HW is not owned by you, it can be a problem. If that's a problem you can just add another layer of encryption to the mix, like a GPG command.

I can and do own that HW. At least as much as one can own anything - it's located in my appartment. You can just as well run the web server from your own workstation/laptop.

The only reason to put shared files on other computer is that the other computer is more available.

Well I can't use this on Windows nor can I guarantee that this cryptic shell script will run on my friends computer who still uses Windows. Even so, he wouldn't know how to run it.

Also it can't be a "secure sharing service" if you're still using 'sha1sum' and it not being end-to-end encrypted...

So use sha256sum, if you don't trust SHA1.

Also it's quite secure since file content is clearly mapped to the URL you're sharing with the other party via independent channel and the receiving party can verify the content of the file.

It may not be completely private if VPS does not run on your own HW, but nothing prevents you from running it on your own HW in your secured location.

You can use this on windows the same way you'd use it on Linux. You just install a bunch of programs and run commands from command line. (basically just like you'd need to do with fsend) It's less hassle on Linux, since the commands are pre-installed. Fsend is more hassle on Linux since it's not preinstalled and requires rust, compilation, etc.

The other people just open a https links and download files in any browser or via curl, no tools required at all, not even JS.

Great Dropbox replacement.

I might not be that interested in using it, but now I might be interested in reading a bit of the source.

Prevents people from forking and submitting PRs, given rust's abysmally low usage, and higher complexity for beginners.

Or with python just going to dir you want to share:

python 2.7 >python -m SimpleHTTPServer 8080

python 3 > python -m http.server 8080

That's not the same thing, Firefox Send stores the files on the cloud, it's accessible from anywhere without requiring your computer to remain accessible.

You can also recreate Dropbox just by combining an FTP Server with curlftpfs and SVN :)

Python’s built-in HTTP server is pretty useful. It doesn’t really overlap with the functionality of use-cases of a Send client much at all.

Reply on news.ycombinator.com