Happiness is a freshly organized codebase
slack.engineering
slack.engineering
I cannot even get started in projects without a sane repo layout. Source files scattered everywhere unrelated to each other, utils junk-drawers with dozens of files, multiple top-level source directories without explicit rationale, tests and source totally disjoint, hacks to modify build and run paths, implicit dependencies between directories, awful convoluted build-system configuration to match all of these idiosyncrasies, and impossible or very difficult editor/ide integration as a result. Even rails-eque apps (which come with a reasonable structure) get really messy really quickly if somebody doesn't have the diligence to stay on top of it.
I want there to be a {sane, polyglot, large-ish-scale} project layout convention that plays well with {intuition/discoverability, build-systems, editors, project size}, but it seems there's an inherent tension between these. Maven tried. Kinda. I wonder if there's a fundamental problem we aren't solving somewhere.
Caveat: I use IntelliJ and I just jump to tests using CMD+SHIFT+T shortcut so not necessarily a generic example.
Working in chaos is just masochism. There will be enough chaos you can't control, don't add accidental chaos to the system.
(I do get your meaning, I just found your phrasing a bit funny.)
There were only two 'adults' in that shop - the manager, and Bob, the senior mechanic, and I'm pretty sure both of them were maybe 28. The rest of us were college age. He was constantly irritated that we couldn't seem to keep our shit organized. Most of the time this was some muttering or making little jokes at our expense.
Until you wanted to borrow one of his tools because you couldn't find yours, then you got an earful.
I was pretty sure I didn't like Bob at first, but now I'm not so sure.
I see so many similarities in the jobs. Great mise-en-place makes a great cook just the same as a great dev. Remember though, even the best cooks have to chop garlic during service occasionally.
Consider a file like "package foobar; func DoFoo() { ... }; func DoBar() { ... }". You then refactor that to two packages, "package foo; func Do() { ... }" "package bar; func Do() { ... }". The body of foobar.DoFoo is identical to foo.Do, but no version control system I've ever used will move the history of that function body to the new location.
I guess what I'm saying that tracking renames as a first-class object is nice, but it still doesn't represent many types of refactorings. To smugly declare perfection, a version control system should probably have that feature, because it is a relatively common refactoring.
That aside, there are some many productivity and tooling gains around organizing your project properly, that it is worth history discontinuities.
Also, I've found that the worse the project organization is, the less likely the team is to separate out different fileset changes into different commits to keep a useful history, or make the use of the git history at all outside of looking at a few recent commits or or sometimes looking at the last release. I've heard this as very circular arguments. Can't re-org because we loose history. Can't maintain good history because the project is too disorganized.
Thus, there's a config "diff.renameLimit" and "merge.renameLimit" that tells git to only detect renames if less than N files are affected.
I have personally been burned by this before, and there usually is a default value set in most git clients.
See also some discussion at https://stackoverflow.com/a/7831027/1237375 .
Still good to use the command, just keep in mind git isn't really treating it any different than a normal `mv` followed by `git add`.
https://git.wiki.kernel.org/index.php/GitFaq#Why_does_Git_no...
If maintained properly, visibility settings can stop a bad code structure move in its tracks.
That way, we could have a tool that always correctly shows the dependencies between modules.
In our architecture, we've created roughly 2 different kinds of abstraction: Platform and Business.
The Platform abstractions are only intended to support themselves and the Business abstractions. These are not supposed to be easily-accessible by non-technical folks. Their entire purpose is to make the Business abstractions as elegant as possible. The idea is these should be slowly-changing and stable across all use cases. Developers are only expected to visit this realm maybe once every other week. We effectively treat ourselves as our own customer with this layer.
The Business abstractions are built using only first-party primitives and our Platform layer. Most of this is very clean because we explicitly force all non-business concerns out into the Platform layer. Things like validation logic are written using purely functional code in this realm, which makes it very easy to reason with. Developers are expected to live in this area most of the time. Organization of business abstractions typically falls along lines of usage or logical business activity. We explicitly duplicate some code in order to maintain very clean separation of differing business activities. I've watched first hand as the subtle variances of a single combined Customer model across hundreds of contexts eventually turned it into a boat anchor.
As a consequence of all this, our project managers and other non-code-experts are actually able to review large parts of the codebase and derive meaningful insights without a developer babysitting them. We are at a point where project managers can open PRs for adjusting basic validation and parameter items in our customers' biz code. Developers also enjoy not having to do massive altitude shifts on abstractions while reading through 1 source file. You either spend a day working platform concerns, or you are in the business layer. Context switching sucks and we built to avoid it as much as possible. IMO, having a bad project structure is one direct side-effect of being forced to context switch all the time.
The crazy thing I've come to realize is that the journey doesn't have to end there either... You can build yet-higher-order abstractions on top of your platform layer (i.e. a platform for platform). I don't know where this all ends up, but 1 or 2 iterations of it has been extremely healthy for our architecture and business use cases so far. We are now able to chain together extremely complex business processes in ways that can be reasoned with in a single POCO mapper. Without a separation of the "noise" of platform-related and other lower-order code from the business code, it would become impossible to see these opportunities.
But I end up with a strange mix of platform features and business-case features. I can't believe I never thought to simply separate these too distinct types into different parent folders...
$ git config blame.ignoreRevsFile .git-blame-ignore-revs
[0] https://github.com/git/git/blob/master/Documentation/RelNote...
Perhaps one mitigation could be to include a message describing something like this in the git commit for the mass-move.
In my experience, ensure you don't change the code as you move it, don't rename files and move content at the same time, and don't move "too much" code in a single commit. Around 2k lines at a time seems to be a good number. Maybe some languages/structures are easier for git to analyze when doing blames.
> No more dumping ground folders like “Helper” or “Utility”
If you’re organizing by feature and you have one of these helper/utility classes that support multiple features, where does it live? Would you consider each utility to be its own “feature”?
Keeping that in mind, if a feature has a shared helper across components within the /Feature folder, we'll have a /Shared folder within to capture these files. At a top level, we would encourage a new /Feature folder since most likely it will have tests or be important enough to merit a folder rather than squishing it into another /Feature folder. We're optimizing for visibility too, so having it's own /Feature folder helps with that.
If you work at an agency, or game company shipping many games, there will always be a "core" library or "base" library that has Helpers and Utility. Good consistent projects wanted tested and solid parts. Every game has a common lib of core tools like maths, vector tools, prediction, data structures and many more.
There is such a broad difference in coding on one platform for one company for one product compared to shipping many companies (or many internal projects) on many platforms for many products.
Even standard libraries for platforms are essentially a Helper/Utility really when it comes down to it: .NET Core, Python standard lib, standard node, C++ distributable etc. There will be common core tasks across a project, product, company or platform for most teams with many projects to manage.
If you don't have common libs with common helpers/utilities in large sets of products, technical debt and maintenance become a nightmare. In projects we work on these are core/base libs that are submodules in git and are essentially the 'tech' across projects where the project/product itself is the unique implementation for that product. Anything common or generic gets put in the core 'tech'. Every single game studio will have these as well as agencies if they are organized and produce quality relatively fast and consistent.
If "Happiness is a freshly organized codebase" they must have common tested parts that are ship tested.
This is really just Slack talking specifically about mature products at a company that only has one product. It isn't reality at places that have many projects, products, companies, clients etc. Pretending they are the same is a bit elitist.
Helper - function which makes some really common project specific code more DRY (createApiError, genCommonConfig, etc)
I’d say I’d expect a “utility” to adhere to these conditions more strongly than a “helper,” which might be a bit more entangled with application logic.
Yes, but in my experience this is rarely done. It's put off until it gets so convoluted that it introduces bugs. The same is true even if you do avoid dumping-grounds: you need to examine the project's state somewhat regularly to ensure it still makes sense. Where code lives can be as important as the code itself and thus deserves at least occasional review and change.
I believe it's totally okay to name things utilities when you haven't yet established a common pattern or understanding of it. Naming things is hard and spending so much time on it and organizing things can often block you from progressing to a place where you DO have more information and can intelligently name things.
You'll always be juggling unknowns, so it's okay to have dumping grounds here and there, so long as over time you clean them up as you gain more info.
It was brought to my attention by this post https://csswizardry.com/2013/04/shame-css/
And yes, there needs to be a structured and scheduled review and cleanup otherwise it becomes a nightmare. We do this every 3 months along with scouring the codebase for TODO's and FIXME's. It's done in mob-programming style with drinks which makes for a lot of fun and a good way to share our "mistakes". But to be honest, it's starting to devolve into a "coding roast" of sorts!
Juniors could try out new things and add in naive fixes for bugs. Then in code review we brought the change up into the cascade as a method of teaching cascade thinking.
It's rare to do that now as most front end devs are not interested in the cascade and use things like BEM instead. I can understand why - it's pragmatic - but I did personally find that a great way of building up devs who really grasped css on its own terms.
It's very possible to wind up with massive comment dumps of things to research, alternate implementations, notes to yourself and other things littered in your code base where you haven't made any git commits yet.
This really leaves things in a messy state where you feel like the project is never going to be finished.
An example of this and how I solved this problem with a kanban board can be found here: https://youtu.be/HHOkcCqsipE?t=76
For example, when I write a library its usually very simple. There's a single directory which contains all of the source files. When people use the library they:
"""
import lib
lib.run()
""""
Dead simple, no complex module paths to remember, no hierarchical folder structure forcing you to code based on a pattern rather than functionality. Pure bliss.
But on the other hand, I have projects that contain 100k lines of source code. I can't just leave it out in the cold. So poor baby gets a couple of folders.
But I do hate it. I hate writing the code. I hate reading it. I hate finding it 6 months after the fact.
That's probably just the nature of the job. It is work at the end of the day. Maybe its just doomed to be hard.
I structure larger projects as libraries with minimal dependencies that depend on one another, and dump all my modules with descriptive file names under the same directory within the library.
I vaguely remember reading something that hinted at Facebook doing something similar with their React components.
It visualizes dependencies in a project. I found it so so easy to refactor and move files around after I started using it. I am not usually a visually-oriented person, but for this usecase, and to be happy, `dep-cruiser` surely helps.
You'd just have a "new code block" button, which creates the editor tab for your code, usually a function or a class, and usually one item per block. When you save it, it puts it into a database and you can version things easily. You can call other functions from the block and your editor will show their code when you mouseover or maybe some other method, just like today. Basically the same as today, but you don't need to worry about where some code lives. You back it with a great search feature to find stuff.
Hell, let's just eliminate pathed files, why do we care about file paths with the level of search today? Just store everything in a key:value store directly on the hard drive, no paths needed. For legacy, just add keys for '/etc/fstab' or whatever.
An extensive, unrefactored codebase is no different than a jungle.
You might have a 10'xer programmer on your staff, and he might hold the programming equivalent of a machete, but if the rate of his refactoring (assuming your corporate rules let him) is slower than the rate of new code being added by other employees, he is going to fail, no matter how good he is!
I need to write a future essay about the relationship between 10'xers and how a 10'xer is a combination of only as good as how well the codebase is refactored, how well they know the codebase, how much corporate rules/polcies permit refactoring (or not), how much time he doesn't have to waste time solving stupid one-time issues from single customers, and how much help or pushback he is or isn't getting from the rest of the team.
In other words, given the right set of conditions between codebase size and obfuscation, limiting corporate policies (i.e., "you can't refactor", "you can't make just one mistake in your code, because it's all mission critical, and if you do, you will be fired, and by the way, there's no test environment!"), distraction ("you have to help this customer with his cosmetic problem before you are permitted to tackle the guts of the system"), and pushback from the rest of the team, you can actually change 10'xers (and higher!) to 1x'ers and below...
The reverse is true too...
I'll make anyone a "My Fair Lady" / Eliza Doolittle style bet (or the reverse!) that what I say is true!
That is, that 1'xers can be taught to become 10'xers, and conversely, 10'xers can be hampered by a variety of factors ("The Perfect Storm") resulting in them being slowed to 1'xers, or below...
You mean like the post-commit hook that Git offers out of the box? It's even named the exact same! I feel like we don't focus all this time on fast build test and deploy cycles only to then commit, navigate to some website to create a "merge request", wait for some fairy to allocate computing resources for trivial checks my computer could have done, only to get some pure noise "don't put this file here" comment and repeat the cycle all again.
Surely valuable, but to say it is actually a Go standard is misleading IMHO.
These are all done by people that need to feel in control and that everything has a place. So much money and time has been wasted on stuff like this with zero proven or measured benefit.
If having a codebase that's well organized makes developers happier, and happier developers stay longer at the company, then that's also a win.
Organization for happiness is based on the individual developers personality and means you are putting some engineer’s happiness as a priority over spending that money providing other things to make the developers with a personality type that derives no benefit from strict organization.
In fact, strict organization rules make some developers less happy. Why is their happiness less important when this is well known to have no business benefit?