Building a BitTorrent client from the ground up in Go
blog.jse.li
blog.jse.li
> For brevity, I included only a few of the important snippets of code. Notably, I left out all the glue code, parsing, unit tests, and the boring parts that build character. View my full implementation if you’re interested.
Any chance you'll do independent write ups to cover all the missing bits? Especially the ones that build character. :)
Wondering if there are any similar docs or a book for implementing a server in Python, and a client in Python or Dart/Flutter? I know libtorrent bindings are a thing but the docs seemed quite dense and I didn't even know where to start—until now that is.
Glancing at the wiki page for BitTorrent[0], I'm a bit surprised that there isn't more of an effort to create cross-platform libraries and clients using Go or Rust for this. Seems like a perfect use case? For example, synapse [1].
[0] https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clien...
[1] https://ipfs.io/
But as a streaming technology, torrents are still pretty great. And like it or not, being able to run in browsers is a major advantage right now.
The case the author is going after is downloading a large Debian distro, so in the use-case of large, distributed file distribution there will always be solutions like the author.
1. video streaming
2. centralized sources like news websites
3. sharing in group chats
End users don't really download large files like Debian outside of gaming and OS updates. To that end, Blizzard Downloader [0] uses BitTorrent, and Windows 10 uses some similar p2p system [1]
[0] https://wow.gamepedia.com/Blizzard_Downloader
[1] https://www.pcworld.com/article/2955491/how-to-stop-windows-...
Donating bandwidth to a community torrent for the general good is not really the same thing as donating bandwidth to a for-profit corporation. I'm much more inclined to do the former than the later.
You close browser = close WebTorrent = you stop seeding
You stop watching video = you stop seeding it
Long and large video file? = if most people watch at half of video, then data from beginning is already out of the buffer = no seed for first half of video.
To this day there is still no support for WebTorrent in other popular Bittorrent clients so they can't connect to each other. But there is some movement in libtorrent library so we will see.
A quick glance at the webtorrent readme says it uses UDP on node.
There is a webtorrent desktop app in beta that does both.[0]
Here's hoping it makes it to libtorrent. Even then, I'd imagine it would take quite a while to make it into the clients running on seedboxes, desktops, etc.
It's more convenient, it's safer, it sucks for other torrentors because I don't seed as long, but let's be honest, most people downloading torrents don't care.
Of course, you could make the personal decision to trust a client, and that is fine. But if you aren’t willing to blindly trust a client, the other guy’s point still stands - browsers are probably just the better choice here from a security POV.
And as the original article demonstrated the first half of that is a weekend project.
They also have a monstrous attack surface because they are "web-facing". A specialized client that only implements one protocol without any connection to the "web" is far easier to reason about and debug.
If you only consider the number of man-years an application has been battle-tested, you imply that design complexity and attack surface doesn't matter. If we account for complexity by using a metric like "(man-years of battle-testing)/(magnitude of attack surface)", a well-tested specialized client that hasn't had many recent bug reports is a much safer choice than anything running in a browser.
> blindly trust a client
That's even worse for the browser: you have to trust several orders of magnitude more code implementing a massive set of interdependent features. Yes, there are probably a lot more people working on fixing bugs in the browser, but there are also a lot of people adding/modifying features and thus creating new bugs.
My sibling who isn't a programmer. They would have no clue what is trustworthy or not. Downloading random executables from the internet is not a good idea. How should they know qbittorrent is safe as long as it's downloaded from www.qbittorrent.org but utorrent is basically malware? (Or at least was for awhile).
They already know they can trust firefox and chrome, so it's better for them to just do that.
That's before you get to questions about security of the clients, security of the website you are downloading the torrent clients from, etc.
OTOH, `apt search torrent` on Ubuntu probably doesn't recommend any malware. Though their GUI nowadays promotes snaps and I'm not sure how much that is better than random executables.
If you’re worried about how “sketchy” it is most busybox applets can be read through and totally understand in a few hours tops.
But you're missing the point, even if it did it wouldn't be a torrent client non-technical users can use. It's not me, a programmer, who I'm talking about here.
https://github.com/DiegoRBaquero/BTorrent/issues/71#issuecom...
I'm much more concerned about "real" viruses like ransomware that the browser does successfully protect against.
For a dedicated torrent client to infect your PC with a "real" virus, you'd need to download the torrent and execute the file yourself (PEBCAK). I trust my own judgement much more than some random webpage.
Or in other words torrenting becomes no worse than everything else you do.
Someone is actively working with the maintainer of libtorrent (popular implementation of BitTorrent) to get support for it into the library: https://github.com/arvidn/libtorrent/pull/4123
[0] What are those? Plain Old TCP Sockets?
[1] You mean, WebRTC, right?
This would obviously require majority of the torrent clients to implement WebTorrents as otherwise it wouldn't make sense and you'd exclude more than you would include. If I'm not mistaken, libtorrent is used among many of the open source torrent clients, and so I think merging this code would suddenly give WebTorrent users an enormous amount of peers once they all upgrade. For client implementations upgrading is probably requires next to no code changes.
Also running things in a browser is at cross purposes with torrent long-term availability, large-scale (giga or terabyte) file management, efficient IO and so on.
I think the focus of the class was roadmaps and other parts of the development process for a larger project. Most of the groups picked games or things like that. We chose to build a bit torrent client and I ended up continuing working on it for a year or so.
I was pretty proud of it at the time. I'm sure there are a bunch of implementation details that seemed like a great idea as a Sophomore in college that no longer sound so clever, but the implementation is pretty sound. I used it to download a lot of stuff over the years.
The part I think is most interesting is the code for managing the torrent pieces and writing them into the proper places for the various files within the torrent. (Not necessarily my implementation of it, just the math behind the process) [2]
I remember feeling really clever the first time everything came together and actually downloaded a full torrent. My first bit software development victory.
[1] https://github.com/war1025/Torrent
[2] https://github.com/war1025/Torrent/blob/piecereg/tcl/tm/torr...
That's great to hear --- I know there are quite a few developers who would do lots of different projects, but only to the point of getting something that barely works, and then never use it again. Having it in regular use means you have to continue to fix bugs and make improvements, which a lot of developers unfortunately seem to hate doing, although it's often the more valuable skill. "A true craftsman makes his own tools," as the old saying goes.
Can you explain a little bit how this is interesting mathematically? On first thought, it seems like you would probably use a lot of pwrite, or if want convenience over safety, mmap all the files and write directly to memory. Not to diminish your achievement, but I didn't immediately see anything interesting here. Can you explain a bit more?
The way the files are laid out for a torrent, they are listed in some order in the initial torrent file, along with their individual lengths.
The data for the torrent is then treated as the contiguous sequence of bytes for the files in the listed order.
That contiguous sequence is then broken into "pieces" of whatever fixed size. So finding which file(s) and what position within those files piece 148, for example, goes to involves some math that is pretty straight forward all in all, but is still rewarding when it's the first time you've done such a thing.
Be careful with using this client --- the swarm for a Linux ISO is probably a bit more forgiving, but you may get banned very quickly by the tracker or other clients, because they will definitely notice that.
Initially it was a project to learn about the Core/Async libraries. I will put it back to life at some point, but it grew quite big and became very time consuming for a single developer.
I heard cryptopals [0] is really good for learning both crypto and a new programming language. It might be cool to build things like a DNS resolver, an X window manager, gameboy game, or something from coreutils like grep or tar
I wouldn’t recommend writing a recursive resolver as a weekend project, to many edge cases that you’ll get frustrated by, but that said, I do use writing an authoritative DNS server as a great way to fully learn a new language. You get file handling, grammar parsing, serialization/deserialization, bitpacking, network I/O, data structures, both tcp and UDP socket handling, deamonization, concurrency, etc etc. Covers most of the more difficult parts of many languages and once you’ve done it once or twice, doing it again for new languages is less about figuring out how to do it, but rather how to do it in X language. At last count, I think I’ve done upwards of twenty languages now.
`copy(buf[1:], h.Pstr)`
In this line, are you copying the entire buffer to a string? Doesn't it overflow into other data elements?
Also, and I may be wrong, in the following line, it appears that you are casting to a []byte when it's already a slice of bytes, which should still be fine.
`peers[i].Port = binary.BigEndian.Uint16([]byte(peersBin[offset+4 : offset+6])`
I really enjoyed the tone and the code. I'm not done with the article, but I love it so far.
copy's arguments are like copy(destination, source) so we're copying the string into the buffer. Also, copy will never overflow because it will only copy up to the length of the shortest buffer.
you're right about the unnecessary cast to []byte -- that function used to take a string as arguments, and when I changed it, I didn't change the rest of it. I've removed it.
https://github.com/webtorrent/webtorrent
I guess the VM part is just to use a unikernel then try to find a way to auto use vpn.
That said, quite famously, Bram Cohen, the co-inventor of BitTorrent, failed the Google interviews.
[0] https://news.ycombinator.com/item?id=18711980
Btw, here's a ref: https://web.archive.org/web/20200105225938/https://www.there...