What we learn from twitch source code leak
everlastingbits.com
everlastingbits.com
This most likely was a temporary solution that grew over time and was left like it, they just kept adding new combinations. It works. If you want to add a new combination you just add it and run a migration (or similar, I've not worked on Twitch but I assume it's not the most difficult thing in the world if it's still like this).
I understand that this can feel like an itch that you really need to scratch (I also feel it) but if you literally see the source codes of any big company you'll find stuff that anyone can criticize from their chair with little effort. Let's now go build a tremendously successful platform like Twitch without cutting a single corner ever and see how that goes.
Really bad code grinds progress to a halt when developers have to spend all of their time fixing tech debt or working around fragile codebases to get anything done. However, I don't see anything in this code that would match that description. In fact, anyone can take one look at this and know exactly what it does and exactly how to modify it.
Counterintuitively, perfectly good code also grinds progress to a halt if developers become too focused on doing things the "right" way instead of shipping reasonable code that works. Developers who get lost pursuing a platonic ideal of the perfect code will be perpetually disappointed inside of real-world constraints. Perfect is the enemy of good.
It's definitely not fair to criticize a single snippet extracted from an entire company's code base. It's turning into a cheap way to dunk on a company from the sidelines while ignoring that the company has done a good job of scaling a video delivery platform and community to a massive number of users.
This code doesn't feel like the latter.
> The programmers who made that probably know all of that, or at least that their implementation isn’t ideal They made that not because they are stupid or unprofessional, rather because they are employees working under pressure coming from their managers, bosses, and their deadlines. What was suggested as a “temporary” solution, turned into the permanent.
Your comment is making me irrationally angry. This code example should be lambasted into eternity.
This should have been converted into a database table many eons ago. By doing so, it would save on stories, testing, builds, and more. They would be able to be more reactive to changes and wouldn't have had to push through a new build to make a single or multiple changes.
If they hadn’t so egregiously failed as a technology company and betrayed their user’s trust then maybe people would have more empathy, but no, they pursued profits and ignored security and stability. I have no empathy for this.
This solution is simple and good. A developer on the receiving end of a "add an item to this list" bug report has a clear, well-trodden way to do it: update the text file. They have a modern editor that handles big files just fine. They frequently deploy new code to production, including testing it.
Moving to a DB means that it needs an interface in production, because presumably Twitch doesn't let their devs run random commands on production boxes. It means ACLs and a sync between dev and prod. It means another moving piece when Twitch spins up a disaster recovery test.
The full-powered, industry-standard solution is an AI-based spam detector with some sort of lag on blocking (e.g. shadow banning). This requires inputs to train on, such as reports from Twitch moderates, which might need oversight. It requires ML engineers and quality heuristics. This stuff is getting cheaper, but it's a hell of a lot more expensive than an ugly text file.
One company I worked at had poor code quality, but I chalked it up to poor engineering culture - a lot of outsourced crap code and no time dedicated to go back to clean it up.
Another company also had a code base with terrible quality but I figured it was because it was a startup and they were rushing to get features out.
None of the companies I've worked for are big ones, and I've always wondered if these levels of quality were the norm, and leaks like this really give me insight to what would otherwise be private code.
Of course, Twitch was also a startup so maybe this was a hack that made it into production as the author notes, although I don't imagine a better solution would've taken much more effort to implement.
Well you could also argue that I'm the common denominator here and the code isn't poor per se, it's just that my expectations were too high when coming into the professional scene - which is what I'm leaning towards currently - I don't really know what to think.
In the rare cases where you do see elegant, clean, high-quality production code, it is usually because 1) only a very small number of people are writing that code, 2) there is a strong cultural norm of rigorous code hygiene among those people, 3) they do not have prohibitive time pressure to ship, and 4) the requirement to maintain backward compatibility across versions of the code is weak. In practice, these conditions are very rare. I've mostly seen it in what is essentially hobby code, where the craft was a large part of the objective (and hobby code that becomes production code tends to quickly take on the characteristics of other production code).
There simply aren't enough people that care about code quality for its own sake, and the economics of maintaining very high-quality code rarely makes sense in practice.
For me elite just might be a company that meets my standards, which I realize is a useless description for anyone that's not me now. In fact, it's a lot of personal opinion - mainly code that's easy to modify and reason about I guess.
I realized I said "this bad" in my original post but, to be clear, I don't have any serious issues with the Twitch snippet - I meant "this" as software in general. The only nit I have is using `like any array(['%p1%', '%p2%'...])` seems like a more elegant and straightforward(?) way to write the logic.
This is an amazing read on the software controlling the space shuttle from 1996. Thanks for sharing!
Wow, people have problems. Does it work? It does. How often is this executed? On sign up and maybe nick change. Is it encapsulated? Yes, it is. Move along, nothing to see here.
It is not. You've misunderstood that code snippet.
> This whole check should not even be in SQL.
> Similarly, for our original problem we should have a function or class which purpose is to check whether a word matches against many regex phrases.
Interesting conclusion. A Very confusingly-worded article.
Facebook has a list like this. So does Twitter. Google has several and an ongoing initiative to consolidate them behind a shared service.
The login names are very specific, despite the use of 'like' and wildcards. It looks like an exception list that is added to as login names are chosen, spotted, and banned. Which means someone just pops into the code and adds a new entry as a 'bad' login comes to light. It isn't expected that it will proactively block bad ones but that it will stop already-used bad ones from recurring. Which explains why it looks as it does; it has just grown from a quick hack way back when.
- About the Code Quality
I'd guess none of us would set out to write it like this if we knew how it would grow, but for those original handful of exclusions a database would have been overkill.
There comes a point where you start considering refactoring, but in a business with priorities it isn't unusual to look at code like this and say it's encapsulated, efficient, version-controlled, and in just one place, so other things are more important than changing what isn't actually broken (meaning the location of the code/data; I'd probably have issues with the actual logic but that's a different thing).
- And the Straw Men
As for the article this thread is discussing I'm not a big fan. The second half says "This whole validation should not even be in SQL". Yet the only reason that's even up for discussion is because the first half suggests it. The code certainly didn't. Which makes most of the article pointless.
And the reasons to complain include "how can you manage that txt file of or statements?", "What do you do if you want to change it to something else?", and "copy paste each phrase in a 1000 lines?".
It's simple - you manage the txt file (actually a file of source code, but whatever) by editing it and versioning it. Want to change it? Then change it. There is no reason whatsoever that it would result in copy/pasting 1,000s of lines when only the ones changing need changing.
- And the Overhead of the Proposed Change
A tool? To maintain a database list? Why waste sprint time creating a tool to replicate basic textual operations in code that is simple, isolated, and fast.
- Summary
Sorry for the rant. There's good code and bad code, but there's also good enough code and whilst as I said I'd never set out to write this there's no real reason to change it.
YMMV. Five developers have five opinions plus a sixth for the consensus. We won't all agree. I can live with that.
If the Twitch codebase hadn't leaked, how many people would have known that their naughty-word-blocker was implemented in a less than perfect fashion?
Also, any large codebase will develop some wharts due to circumstances you can't see when simply reading the code.
The way I see it, programmer perfectionism comes from learning - the projects you do for yourself, early on, are small enough that you can hit perfect trade-offs on your limited needs, and you aren't under time pressure. You quickly learn you can achieve near-perfection - a lesson which stops applying when working with any nontrivial codebase at work.
(Not sure where I was going with this)
You can add a lot of spice by propping up the care with some nicely tuned turbo, pedal behavior and cooling that is sufficient for just a few accelerations. Most people will never find out the difference.
This snippet is not the biggest problem with the twitch leak, I spooled through the terraform and a lot of it is awful. But it obviously works for them.
I had a overwhelming realisation that code quality really doesn’t matter, I spend a lot of time making my code clean and easy to understand- and that might improve _my_ ergonomics. But realistically my company is not bigger than twitch and if they can operate without major problems with this code then obviously it doesn’t matter.
- it's got a history/blame with associated bugs linked in the changes, including all the reasoning behind every line.
- it can be versioned into rollouts and rolled back if it interacts with another system in a bad way.
- it's dead simple and self describing.
What's not to like?
- Bad code lives in the dark. Code that is visible by many others never looks like this.
- Shame is good to some extent. It keeps people accountable.
- When closing tickets is more important than actually making real improvements, code looks like this.
As someone who has attempted to make a modification to OpenSSL, I feel like this claim needs a big old "[citation needed]" banner.
OpenSSL is known to be complex, hard to audit, etc. But they do have standards.
You may want to take a look at LibreSSL or BoringSSL instead if you are looking for something more modern and maintainable.
There's also the parameter parser builder at https://github.com/openssl/openssl/blob/master/crypto/param_...
This kind of pattern of repeated application instead of iterating over a data structure ends up fairly common in any code base I've ever seen, public or private. The machinery to build a list loader and iterator always just seems like more code than one wants to write when one is tasked with adding one more element to the existing sequence of homogeneous operations varying the data.