The Messy Page Or: why I don't like greenfield projects
registerspill.thorstenball.com
registerspill.thorstenball.com
If we were to rewrite this thing from zero with reckless abandon, all of that confidence & trust that built up over half a decade would be lost. We are in the process of investigating alternative architectures, but we are doing it in a way where we minimize how much we have to involve our customers and other team members. Things like "backwards compatible configuration" and "don't change the UI if it works" are headline in our greenfield.
Your response made me think about this - https://randsinrepose.com/archives/stables-and-volatiles/
One of the ways greenfields go wrong is confident people bringing too much to them. They're sure that features X, Y, and Z are vital. They're sure that the best technical approach means architecture A with framework B and library C. So they jump in and build things, ignoring the lack of product-market fit and the broken feedback loop with users. The people really being served are not the eventual customers, but the team itself.
But being uncomfortable with the wide-open sweep of a greenfield project can be put to use to keep the building constrained to what we really know. For a startup I did a while back, we spent the first few months only building disposable prototypes. We wrote only garbage code, and then we put it in the garbage. It wasn't until user testing showed promise that we wrote the first lines of production code. Even then we were very cautious about our technological commitments, because we knew how much we still had to learn about our market and their needs.
I was thinking that they had the right idea themselves when they talked about basic research.
In basic research, there really is no such thing as failed or wasted work unless you constructed a faulty experiment. Whatever happens, it's all added to human knowledge. Finding out that something is not possible, or not possible this particular way, is still finding out something.
If you're thinking that way, then you never worry about wasting work. You're plugging away at a methodical process and none of it is a waste.
But you have to be thinking that the job is to perform these steps and record the result, not the job is to discover a miracle.
Getting back to software, as for the blank page, just how blank is blank? Are we talking don't even know what industry we want service or what? Because if the page is blank, there are many tools and methods, but they are not all the same. So the first job is not necessarily to vomit any code but to characterize the job and narrow the tool selection to those which optimize whatever you're going to need to do the most. Maybe you have to vomit some code to figure that out, but then that was the job of that code was to find something out, not to be a product.
That might be Second System Syndrome:
If everyone just knows, that you probably have to redo things, it looks completely healthy to me. But if it paralysis the team, because nobody dares to start something, because nobody wants to waste work so everything has to be thought until the end before starting to develop something, well, then the discomfort might be too much.
That's the best of both worlds: a reimplementation of an existing project. You have all the freedom of a greenfield project, and you don't suffer from the "blank page syndrome" or writer's block the author talks about. Port the tests over, then rewrite everything, this time with a better architecture. [2]
--
1: Since someone is bound to ask, I am incorporating a Rust standalone app into a larger Elixir monolith. Because while static typing is nice, turning a Rust app into a highly-concurrent and highly resilient with native OS primitive (threads & co.) or, yikes, async, is masochistic when I have the power of the BEAM at my disposal.
2: Tongue-in-cheek here, I am well aware of https://en.wikipedia.org/wiki/Second-system_effect
Not only do you not suffer from the "blank page", you can are privy to all the mistakes made before. Of course, you're now open to make a bunch of new mistakes, but that's ok—there's still the benefit of tons of knowledge from the original project.
Furthermore, the Elixir ecosystem really helps mitigate that "too many tools" feeling.
Put everything in a container, stick it in a dedicated server at Hetzner, and Bob's your uncle.
I pay €35/mo for a 32GB RAM, 8-core, 2x SSD dedicated server with unlimited traffic. Elixir flies on that.
They charge €40pcm for an i7-4770, 32GB RAM, 2x SSD which can be bought for around €100 from any IT surplus (given it's around a decade old) and thrown in a libre/GPL colo at your local hacker space for single digits pcm.
EDIT: just the mobo, CPU and RAM is $300 on eBay. And let me know which hackerspace has a dedicated maintenance team, SLA guarantees, unlimited traffic. Apples and oranges. I'm trying to run a business here, not run a Minecraft server.
Should I really be penny-pinching on what is $50/mo? It's two pizzas, at current market rates.
Designing a robust/performant/reliable architecture right from the start is not "interesting", it is effectively impossible unless you have divination skills. At some point, your architecture will suck, and restarting from scratch is usually a bad idea, and that's when things get interesting. That is, fixing the mess you created, making something robust/performant/reliable out of something that is not, with the additional constraint of not breaking it in the process.
But you still need to start somewhere, even you know it will fail. It means you will essentially do a dice roll and hope for the best. Yes, the beautiful architecture you start with is nothing more than that. If you leave after that, not only it may not be cool for the ones who will inherit your mess, but you also won't get to see the hard (and interesting) part.
Of course, even though it will always be a dice roll, it doesn't mean your initial decisions won't matter. How fast will you fail? (failing fast is a good thing) How fixable your mess is? How will it fair after an army of monkeys tramples over the code? etc... These are questions experience can help you answer, but you won't get this experience by doing greenfield projects. You will get this experience by being in front of the fan as shit hits it.
So yeah, I understand the love for greenfield project. It is like a reprieve, where you can use that experience acquired during the time you were covered in shit, and try to make the best roll. To me, it is the comfort zone, but as often, the comfort zone is not the most interesting.
If you're working on a greenfield project there's a very high chance that your "robust, performant and reliable" is someone else's tech debt once people need different things from the system in a few years... And then the more clever that original work, the harder the rework.
I have a hard time relating. Some of my favorite gigs were where I got to come in and design projects from the ground up. It's where you get to exercise an engineering mindset, where you're adhering to constraints while trying to meet requirements and setting the groundwork for potentially years of contribution, change and adaption.
working on an existing mess with outstanding complaints at least better guarantees that it is a valued mess. some environments, that’s not about comfort, it is about survival.
I can understand why he has such a disdain for government waste.
I think projects which get cancelled or never released are great. I get to do the fun parts - the design/architecture and solving the hard implementation problems - but not the endless of hours of tweaking, bug fixing, dealing with the inevitable pre-release business pivot which turns the software from a beautiful solution to Problem A into a horrible kludge for Problem B. And I still get paid!
A lot of what makes my job enjoyable or not is in whether there’s a business case for what I’m doing. If some VP is asking me to fiddle around with something that’s never going anywhere I’ll be miserable. But if I’m fiddling around to build something that drives more revenue or makes people happy with our product then I’m there.
We had an in-house data centre, and we needed more capacity at certain times of the year. Justification was needed for more dedicated capacity in the face of cloud infrastructure.
So we built a system to use spot pricing to spin up extra instances, securely, on AWS, google cloud or azure. This could meet bursts of demand at busy times.
At the end of the exercise we had a working system and managed to prove it was better value and more resilient for the organisation to increase on-premises capacity.
Fun project though!
I prefer improving things rather than creating them from nothing. I’ve come to a similar conclusion as the author as far as how to get out of the fog of greenfield: Satisfice instead of optimize.
In this context, optimizing refers to choosing the best among N options. In contrast, satisficing means choosing the first option that’s good enough. Often, the opportunity cost saved from satisficing is greater than the marginal value gained from optimizing.
What you describe is a very common pitfall that is completely avoidable using well established frameworks.
In systems engineering one of the key principles is that you build a system to meet requirements. This means that prior to making architecture diagrams or picking your database, you instead meet with all of the stakeholders and enumerate a list of what is required of the system or service you are building.
Once you have requirements, you simply look at the normal trade offs (operational ease of use, cost, license model, performance, etc…) and make decisions accordingly.
There is still some room for decision making, but if you apply a tried and true framework it’s helps cut down on the decision making a lot.
IME “the code is crap, we need to start again” really means “I don’t want to (or worse, can’t) read source code that isn’t mine”.
Reading and understanding someone else’s code then making and executing a plan to improve it is a much more valuable skill.
If you’re always working on greenfield projects, you’ll never understand what it means to build software with longevity.
understood. but sometimes the code really is crap, and possibly a danger to the business.
For example, one thing I look for is when the model can not explicitly express all of the states that we now need, and as a result the code has developed various “hacks” that try and simulate the additional states. This code tends to look like “guesses” or “assumptions”. For example, “if not red than blue” or “if no records than user is still on step one”. I tend to see developers spending hours in that kind of code, trying to confirm that the assumption is still always the correct one to make, or trying to shoehorn in another state, and it’s just a horrible time-suck. Improving the model at this point generally pays off in time saved puzzling.
For example, I’ve seen and participated in the transition of a completely undocumented custom web framework (with an entirely Redis based ORM) to an industry standard web framework with a MySQL backed ORM.
All of this happened without any customers noticing, no feature development pause, no huge “rewrite” branch requiring regular rebasing, and without the original development team responsible for the mess.
That was truly a work of art.
99% is dealing with the after-effects of the decisions made during the 1% ;).
Green field (to me) is a new product entirely. A blank slate, a wonderful opportunity.
Within the confines of an existing company it’s usually better to reuse established tech, for the sanity of your colleagues. :D
That very much depends on the person, industry and company.
We've hired others in the past for special skills - building data schemas, graphic designers, problem solvers, debugging and testing. Because there are many skill sets.
I understand the OP very well. We'd all do better by understanding this!
I hate debugging.
Two big programming barriers I noticed as a kid were: (1) walls of complexity, and (2) debugging interruptions that took lots of time and ruined momentum.
I learned how and when to avert both barriers. And I love greenfield, partly because I can apply what I've learned about both. (And partly because I like working creatively, and with a holistic understanding of the problem domain and approach.)
It's great that there are people who dislike greenfield and love well-defined debugging tasks. I can trade them my brussels sprouts, for their chocolate cake.
Some people are “serial starters”. They have enormous creative fire, feel a burning need to create something new, and once it starts taking form and people start using it, they get bored and move on.
Others are better finishers. They will patiently build out the core until all the required bells and whistles and niceties are there.
It’s very rare to someone good at both, such as Linus Torvalds. Maybe that is a third category, someone who starts a few great things and sticks with them for life?
Another example. I did a proof of concept, demoed it to the big bosses. Got approval for it to be built. I'm not so much at the individual contributor level anymore though. Anyhow come to find the dev team didn't build off of what I started. Built from scratch and spent the first month reimplementing what I had already built. So dumb. Why do we keep doing this in our industry.
Me personally, I love designing a system from scratch. That is a big part of the fun of writing software for me.
You know you can make your greenfield project messy, right?
Over the past few years, I discovered I really enjoy projects that are uncertain, where nobody knows yet how it's all supposed to work and what the end result is going to look like. Projects where you try stuff only to throw it away. I've done projects like that, and they were my favourite projects ever. I love to experiment, crank our a quick proof of concept, only to rewrite it in a different framework and refactor it several times over the first year.
But hey, I guess it's a spectrum that all programmers fall somewhere on. You might be good at starting things from scratch, you might be good at polishing up and improving existing projects, or you might be somewhere in between.
I guess I like the responsibility and dislike conformity.
- The code was basically just spaghetti with differing standards and architectures depending what area you were in. Making even the most simple changes or basic features would often break something else in unexpected ways.
- There were never any tests, and in one company's case, they even had a no comments policy "because the code changed often and they didn't want to maintain the comments' accuracy".
- Said companies did not have the money (or want to spend it) in rewriting any significant portion of the apps, despite them being on years ago abandoned frameworks.
- In one case, this meant we had to patch a web framework and maintain it ourselves because of security vulnerabilities.
I realize maybe the author worked on established but not-so-bad projects, but working on a greenfield is imo the far better experience. You can actually push for standards, create impactful features in a timely manner, and do refactors without fear that 20 other things are going to break because the architecture is fresh in your mind, documented and you have up-to-date regression tests.
The author also seems to primarily be afraid they'll do something sub-optimally and have to delete code. I actually love deleting code. Some of the most satisfying refactors I've done involve cutting down on complexity and removing portions of code. I get that you don't want to waste effort, but I don't think it's a healthy attitude to have toward your code if you get negative emotions over finding better solutions.
As I get older I find myself valuing locality of behavior over most other concerns.
I can deal with some pretty spaghetti-ish code that's bad along many traditional metrics as long as I know what my changes are going to impact.
I absolutely agree, that's a big one for me as well and when I'm designing new features it's always a high priority.
I am much better at adding new features to existing code. The constraints are much better defined. I also always strive for perfect backwards compatibility if possible, which is both great for the users (since they have to do nothing, upgrading just works) and also great for me, since it's another set of interesting and challenging constraints to work against.
I got slightly better in getting out of analysis paralysis by doing Advent of Code in real time. This forced me to start writing code, fast. I think the effect has worn off by now though.
There's at least a third scenario that busts this dichotomy and also isn't the overconfident junior dev nonsense in some of the other comments. I've seen enough of those messy bugs and worked on similar enough projects that are far better put together. I just want to replicate their success. I'll be shamelessly stealing ideas under the guise of "best practices" and I won't worry about the outcome because I already know. The code from me will be easy to read and well documented and nobody will notice apart from the suspicious lack of drama, as it should be.
To each, their own, I guess.
I disagree a bit here. Sure, there are hundreds (thousands?) of databases out there. However, once you add your business requirements, and second+ order considerations (e.g. is this database battle tested in production? is it supported? is it supported in our company specifically?) the choices narrow down dramatically.
In fact I dare say that in most greenfield projects you can probably narrow most tech choices to less than a handful.
my preference is making my own mess and iterating it out of one. decreases the hunt for bodies. but preference doesn’t always equate to reality.
also depends on the project. if there’s a delightfully boring industry standard for most needs, starting from nothing results in a cleaner product. starting with an existing, misguided implementation is just greenfield with extra steps.
This is dead on.
There's a lot of value in knowing where one fits and thrives in the dev process. We need people that thrive building the v1's and we need people that thrive extending the v1's into v2's and beyond. They are both really valuable.
Not that I mind messy pages or projects, I love to refactor and clean up. Just new projects always give you a chance to do really well and clean.
I enjoy designing architecture and protocols, especially when doing so requires me to get down to the bits and even more when doing so requires me to work under tight constraints.
What a weird impulse. Getting rid of a blank page ASAP!! I stopped reading here. Author prefers looking at a page of ads.