If you're interested, the main Bittorrent spec is actually quite small, you can easily read and understand it in an hour or so: https://www.bittorrent.org/beps/bep_0003.html
There's a lot of stuff "around" Bittorrent, like protocol extensions. But in my experience so far, many clients have different levels of support. E.g. they would try communicating using Protocol Encryption (obfuscation, actually) and if it doesn't work fall back to the regular protocol (if configured in the settings).
If you just want to download a Linux distribution (e.g. Debian) using the .torrent file on their website, it's relatively straight forward. It has just one file, there's a tracker, plenty of peers using the regular protocol, and you could get away not listening for connections when you're behind a NAT.
Naturally when you get into the grey area of content, then you'd need to parse magnet links, use DHT to discover peers, and perhaps you'd want to implement UPnP to try and automatically map a port on your modem. But that is also an iterative process, you can just build out one by one. The more features, the more peers you'll be able to discover and have successful exchanges with.