Build your own BitTorrent
app.codecrafters.io
app.codecrafters.io
Some back story: After being laid off from my FAANG job, I found myself very unmotivated to go back. I started looking for interesting programming projects to revive my interest in coding. While nomading, I discovered Codecrafters on Nomadlist and really liked the push code to git and pass different stages interaction. The gamification helped me focus and projects allowed me to go deeper on software I used (SQLite, Git, Redis etc.). I even picked up a new language (Go) to do the challenges with. After completing all the challenges on the site, I ran out of things to do. This is when I decided to build a BitTorrent client which was one of the highly voted ideas on the site.
I learned many new things by building a BitTorrent client: the BitTorrent protocol, how torrent files are structured, encoding issues, pipelining network requests, url encoding binary values, using Channels in Go etc.
I’d love any feedback on the challenge. Also happy to answer any questions!
Clients also self-report the amount of data transferred. That's not great in a community that fetishizes share ratios. I've heard an op say "there's no excuse for having a ratio less than 1", which makes as much mathematical sense as the parent who told my (math) teacher friend "this is [private school], no student should be below average".
You can theoretically verify upload/download numbers because the total amount uploaded in a swarm should equal the total amount downloaded, but there are all kinds of reasons why the numbers wouldn't match. Maybe a client lost connection and couldn't send its final announce. Maybe one client is sending bad data (I'm not sure how that is reported, might be implementation specific). And clients only send transfer total updates when they connect to the tracker to change status or request more peers, so every client will have a different degree of staleness.
Even if you can tell that someone in a swarm is lying, who is your culprit? As long as they're not being egregious, there's no way to tell.
Maybe the context of the quote was in regard to a private tracker?
The "ratio" in terms of private trackers isn't always the real ratio of GBs uploaded or downloaded.
There are for example some private trackers that grant additional credit for longer seed-time or they declare specific torrents "freeleech" so they don't "cost" ratio.
In the end they are just some of the measures private trackers take to strengthen their network, but they lead to a confusing definition of "ratio".
Are you employed by them now?
If this is a concern of load/execution times on a remote builder, it would be cool if I had some way to run the testcases locally to avoid this concern
I'm planning to add them over time starting with magnet links which is the highest voted extension idea right now
A better link is https://www.youtube.com/watch?v=jf_ddGnum_4 which has chapter marks and has the power outage in the middle spliced away :p
Here are some free tutorials:
JS - https://allenkim67.github.io/programming/2016/05/04/how-to-m...
GO - https://blog.jse.li/posts/torrent/
Python - https://markuseliasson.se/article/bittorrent-in-python/
It achieves the same thing without anyone knowing who I am.
You're right, we could present it as just the articles.
Our overview is publicly accessible and doesn't require a paywall. https://app.codecrafters.io/courses/bittorrent/overview
All the "content" is also available on our GitHub https://github.com/codecrafters-io/build-your-own-bittorrent
If you'd like us to run tests against your code, show you progress, community examples, hints, and so on, then you can do the interactive experience, which requires signing up.
In this Git repository, I see various stages of the source code, and Docker files.
The claim "all the 'content' is also available on our GitHub" clearly does not hold, because at least the article texts are missing there.
EDIT: Sorry, it is there, as was pointed out to me in the answer: https://github.com/codecrafters-io/build-your-own-bittorrent...
They should explain why though
After the handshake you'll need state machines to handle multiple peers, piece-sets to track what you've downloaded, the rarest-first algorithm[0], message processing and so on. The final result of having a downloaded file falls out of that almost as a side-product at some point.
The code base is delightful to read. A lot of developers are better at writing code than they are reading it. If you're wanting to flex your code-reading muscles, I haven't found many better places to start than the WebTorrent codebase. I put it up there with redis in being fun to read.
It's probably my favourite way of learning a new language, as it's simple enough to understand and implement
Hadn't heard of CodeCrafters before but I love how academic their challenge ideas are (eg build a DB). I'd love to see a compiler build in there too.
https://gitlab.com/webai-open/network/interview-challenge
Take the guidelines to heart though. We evaluate you on demonstrating understanding of what you did, not that you completed the course.
My advice for standing out would be to continue building on it past the end of the course and do something cool yourself.
Our salary bands are competitive outside of FAANG (my previous life was at NFLX), we are fully remote, nomad friendly, and work 4 day weeks.
You can come on full time or as a contractor (your choice).
We give space to learn and do things right. Are comfortable investing in knowledge today to see compounding returns tomorrow. For example, I spent the first 3 weeks of my employment here sitting on my couch reading research papers. That paid dividens, we collapsed a 1.5 year timeline into a 1 month timeline. 3 weeks of reading research papers and 1 week of building got us to a milestone we didn't plan on reaching until a year+ into the project: we trained a model running on a developers laptop in Grand Rapids Michigan against a dataset sitting on another laptop in Yorkshire, with a fully auditable CI/CD log of what data was fed into the model.
Another example, the team decided we should do rust to cross-compile to WASM etc. from day one, so we all took 2 weeks to study the Rust Book and learn together. Now we have a subset of our p2p stuff compiling to WASM and running in headless firefox during our integration tests from day one.
Pretty flexible in every respect, just need good people who can help build this.
The network is fast enough for lots of small files to not really justify it.
It seems you would have essentially unlimited bandwidth.
I personally suspect that having centralized repos is a legacy technology from pre-torrent days when also disk wasnt cheap.
Not really true. Try the following: ssh into the host doing the mirroring, with socks proxy. Now socksify apt through the connection.
The speed difference is amazing. Despite double encrypting.
You are expected to complete the project in steps they define (so for their Redis project, step 1 is to bind to a port, step 2 is to respond to a PING command, etc). If you choose not to pay, you can only complete one step per day, even if you submit code which would pass future steps.
This can be quite frustrating, since each step is often very simple, and IMO discourages producing a well-architected solution which anticipates future requirements, as you're left waiting 24 hours to press the submit button for code you've already written.
Still: It's free, and the restricted progress forced me to not use it for procrastination purposes, so there's that.
Started the build-your-own-git tutorial and there is a hard paywall after the 3rd stage.
There is a "Subscribe" button which takes you to https://app.codecrafters.io/pay but I wasn't savvy enough to notice it or realize what it was.
Only after starting a course did I begin to suspect that I needed to subscribe since there were a bunch of locks all over. However, it was not clear that the locks actually did anything since I could still click on those links. I guess I would have found out after completing the first step and not being able to progress. This appears to be by design, which strikes me as slightly dishonest.
I've prioritised it now tho
The crowd that is interested in these kinds of experiences may also like Protohackers, which is completely free.
Otherwise, you can do the first 2 stages of any challenge without paying. You can also check out all the prompts and overview for all challenges without a paywall, and you can attempt beyond stage 2 if you have the membership.
You might also find tutorials here useful https://github.com/codecrafters-io/build-your-own-x
I have also had positive interactions with Sarp so hoping this product takes off!
It doesn't, but will soon: https://app.codecrafters.io/vote/challenge-extension-ideas?c....
We release a set of "base" stages first, and then work on extensions based on demand. DHT is one of them, magnet links is another that folks have voted for.
And I wanted to learn Rust anyway :)
Not sure what I see. There is no content, nothing clickable. Just decode bencoded strings and integers and some comments.
What's the point?
There’s a few different learning styles out there, but personally for me, I learn best while doing projects. That’s why this service hits the sweet spot for me.
If you’re like me in this way, then I’d recommend it.
The hard part is avoided?
[stage-1] Expected "\"orange\"\n" as stdout, got: "Hello World 2!\n"
This should only affect C# repositories, and we're working on a fix - C# support is relatively new, and looks like there are some teething issues with the caching mechanism we use to run tests (we aim to keep median response times <3s).
Rate limits issue should be fixed now!