The Lava Layer Anti-Pattern
mikehadlow.blogspot.com
mikehadlow.blogspot.com
The new project lead asks management to allow him to re-write the project. It's a 10 manyear task so the management declines.
At that point he decides to do the changes in gradual manner, completely ignoring the fact that it's still a 10 manyear task, while, realistically, his team can spend 2 manmonths a year on refactoring. Thus, the re-write will be accomplished in 60 years.
The same pattern repeats with the next project lead and so on.
"second system" syndrome is very real, but nobody in management ever seems to appreciate "Look, technology has progressed in the last five years since this was written...we can do more with less, pleeeeaaaase let us fix this.".
Nobody looks forward to working with legacy code. The business logic is usually wrong, the business itself has changed, the original developers are gone, the market talent composition has changed, etc.
I'd honestly rather see a business embrace the ablative nature of code and instead build itself with the assumption that, every 2-3 years, chunks will be rewritten. This is an argument in favor of services and better documentation and tests, by the way.
EDIT:
And even worse is when the manager/CTO/whatever admits that the code is going to need rewriting--but not yet. Because that means that a) you're going to be wasting effort temporarily fixing something you know will be thrown out and b) tacitly admitting that they are too scared of change to embrace it and too short-sighted to figure out how to balance business requirements with technical debt.
I've been burned by that multiple times.
A man-month of unpaid overtime for rewrites, not a nano-second for legacy support!
>And even worse is [...] you're going to be wasting effort temporarily fixing something you know will be thrown out
Can you reconcile these two points for me? I fear I may be misunderstanding you. It sounds like you're arguing in favor of treating code as a transient thing, but then lamenting having to write something that won't last.
The subtle difference (not well articulated by me, I suppose) is that in one case, you are given the go ahead to go and rewrite everything to conform to, say, an interface spec or API or whatever. This year's implementation or architecture may be scrapped, but then you get a cleanish slate.
In the latter case, what's happening is you are usually building an adapter over something that exists, instead of throwing it away completely. What then happens is that there is a lot of stress involved ("Is this wrapping correctly?", "Does this still do what it used to?", "Have my tests covered all the use cases?", "Wait, fuck, this original thing never even had tests...so aren't my tests are kind of pissing in the wind?", etc.) and then inevitably the push is to fix the fixes, and then fix the fixes' fixes, and so on.
And sometimes that's unavoidable (maybe the legacy code is, oh I don't know, numerical simulation of a heat-exchanger in Fortan77 that was cutting edge and now the original author is quite dead and buried). But, the painful thing (as mentioned) is when the person in charge even admits that the whole thing will be rewritten anyways.
In that case, why the fuck is the rewrite being put off? It signals a lack of faith in the development team, and a lack of prior planning to deal with downtime or other issues coming up.
And keep in the mind that software industry tends to be on the "fair" side of the fence regarding work/pay ethics everywhere in the world... all other professions are much worse.
And sticking to the subject: lots of programmers I know would unfortunately choose the unpaid overtime instead of the working with legacy code, and, unfortunately, I've even witnessed discussions where the programmers were more or less directly offering to put in the unpaid/unclocked overtime just to convince the management to let them do the fucking rewrite from scratch!
...the world in general is much crazier than you think, and you "crazy talk" can very easily turn into "they way we usually do things down here", unfortunately.
The initial developer doesn't understand, know, or follow basic CS knowledge or design trade trade offs and constraints, but will try to create a reasonable system using whatever has press at the time - Rails, Php + Zend, etc.
Unfortunately they will follow weird aspects of tutorials they googled for studiously while missing out on the design implication of their choices leading to monstrosities like a product display page that requires hundreds of recursively made SQL queries, an entire application where 90% of the code is in one file, database tables that are composed in the most awful ways, and more.
It's this first layer that causes everything that comes after. The root of the problem isn't that trying something new is right or wrong. It's that the original was done so badly nobody can reasonably continue down the same path and everyone who comes after can't figure out how to cope.
Of course it's typically exacerbated by deadlines. When you can add a new dependency and meet your deadline or try to do things the inscrutably bad way they were done previously and miss it (let alone have any ability to test it), it's hardly surprising that people choose to add what they know.
When confronted with something intractable of course you throw your favorite spaghetti at the wall. And of course there's not any sort of order to what sticks.
So I think the lesson is more about the criticalness of doing things remotely right to begin with.
Not mutually exclusive with what you mentioned, but can exacerbate the situation or leave a codebase with even an adept programmer a mess.
It's the result of developer who don't have the maturity to consider the long term risk of their decisions. That has nothing to do with the business leaders.
I freelance which means I see a lot of shit. In any given week I can touch/work with 10+ tech stacks across both Windows and *nix. When you do that on a regular basis it gives you a different perspective on things.
In particular, you write/design code in the same vein as the code around it. It doesn't matter what your personal values as a developer are. Consistency is more important than that. I have been known to pull people to the side and ask them "I'm about to do X, how do you normally do X in codebase Y? I want to do it exactly the way you do".
If you're going to change the way something like a DAL accesses its data you better be able to either do all of it, or you do none of it for exactly the reasons the article stated.
That is purely a technical decision, and the blame for that lies with every single developer who chose to introduce a new tech into the stack for no other reason than sensibilities.
Maybe, just possibly, the second developer could at least theoretically continue exactly from where the first developer left off (a single person or team == a single codebase that's at least somewhat consistent) but it could often make the mess even worse if the first developer really didn't have much of a clue of what he was doing.
The managers are basically the customer who ordered the software and they can't be trusted to know anything. It's a very common case that the customer doesn't know what they need and it's the job of the developers to first figure out the customer's real problem before beginning prototyping.
At this point, it's management's fault.
My group uses github for all of our code and while it's private that might change in the future. If that happens, there are a lot of places where the answer to "why did you do X this way?" is "that's the way they do it" and not that I agreed with it. That's potentially more harm than good if someone was looking at it.
If anything changes it's the engineering "best practices", like unit testing methodologies. But mostly it's just fashion, and who has the biggest platforms and marketing budgets for pushing their framework.
Software development as a field sucks, no one really knows what they're doing. We're all just figuring it out as we go along.
As he says in the article: "We think that the current shiny best way is the end of history; that it will never be superseded or seen to be suspect with hindsight."
I think they could not possibly imagine fitting the whole working set into server's RAM. Which changes quite a lot.
I disagree w/ the GP that it's purely about fashion, but certainly there distinct pendulums which swing back and forth in our field. IMO that pendulum isn't purely driven by fancy alone (i.e. fashion) but also changes in computing power.
For instance the ever driving swing of the local driven/server driven architecture. These days it's called "cloud" but it's gone by other names in the past. At some point the demands of the applications demand too much for the network traffic/network storage/etc and an increasing amount of work is offloaded to the local client. And then, at some point even further in the future there's an advance on the capabilities of remote computing (e.g. network bandwith) and the cycle repeats itself.
Throw away the first prototype version of a software and rewire v1.0 from scratch that you can use as a solid base for future development (for any application that is expected to have a long life - think years, not months) even if it's awesome for the users and works business-wise.
And always write a prototype/MVP first, and plan to throw it away, even if you end up using it in production and it ends up crucial for you business. Yes, you should try and build the MVP in weeks, not months, and you should already have its successor done when you're out of beta... but regardless of this, the pancakes rule will apply: the first one is always awful, and if try to stack the next ones on top of it, the whole pancakes tower will crumble. So throw away the first pancake and start from the second!
Looking at the 10-year-old PHP codebase we have at work, the real cause is the organizational fluid around the project, a lack of accountability/ownership.
Nobody on the business-die can confidently say what ought to be current practice, so the system becomes--via circular logic--its own reference-implementation
Since no end-user can independently state "How We Do Things Here" except by saying "Whatever The System Already Does", that channels the software's future evolutionary paths: You cannot make "common sense" fixes, because the bug might be a feature.
Background: I spent most of the last decade writing, maintaining, and managing a large enterprise app that had to evolve from its .NET 1.1 foundations to the present day. Our code base looked almost precisely like the one talked about in that article, and with the very real problems mentioned there.
But...
1. Sometimes the productivity wins of a new approach are serious enough that it's worth taking the consistency hit. I'd argue that ORMs, for instance, really are worth it compared to the quasi-code-gen ADO.NET layer. Switching to those midway through the project introduced inconsistency, but all future development done with the ORM was much faster (and more reliable) than the stuff done with the old approach.
(And yes, this is a dangerous argument, because it's easy to convince yourself that it's true all the time, if you want an excuse to use a new technology. But if applied skeptically, it really is true sometimes.)
2. Developer morale is a thing. "You can only use technology invented in 2003, none of this fancy .NET 2.0 stuff, and haha LINQ, are you kidding me?" is going to crush people's spirits, whereas "sure, let's try this new module out using ASP.NET MVC" will introduce an inconsistency into the codebase, but will keep developers more engaged and learning new things (and who knows, maybe this experiment will prove to be one of those ones that gives you a big productivity win).
You have to be careful with that, because you can't literally try out every crazy new thing, but you also can't ignore everything new in the name of consistency and expect anyone to want to work on the project.
Those devs are, naturally, in the minority.
But c99 is pretty cool (though as the sibling suggested, c89 is rather common). Anyway, 15 years old, just right.
The strategy will cause a variant of the Cobol effect. You'll either have to pay developers obscene wages to put up with your consistent but antiquated codebase, or try to find developers who are willing to put up with it for normal wages. The latter probably aren't the people you want on your team.
I documented my own answer to this situation here. Basically, don't treat organizational problems as technical ones. They're paying you to do a job, everything else you do is basically an ambition. First rule of ambition is to do no harm.
https://news.ycombinator.com/item?id=8768524
If you're going to refactor, don't refactor in the direction of new. Refactor in the direction of coherency. Do things that you can finish, or, that at least, don't matter that much if you don't. Like adding tests. Nothing wrong with that.
If you want to fix the legacy app, you need to make less ambitious changes, because you need to be able to finish them. Instead of changing the data types, refactor the initialization code. The initialization code runs once, data manipulation code is basically convention and you need to change it everywhere at once or not at all. Refactor it to the point where you can change it everywhere at once and then do that. Or get really good with a text editor and macro the changes.
Giles Bowkett has a great book on it. Unfuck A Monorail For Great Justice. It's Rails-specific but the ideas can work everywhere. Write code that understands code.
http://gilesbowkett.blogspot.com/2013/03/new-ebook-unfuck-mo...
But understand that there are probably better things to do with your time. You can pick out pieces of the legacy app and try to understand them well enough to rubber-duck debug them without having the source code in front of you. Once you have that then really, you don't need to be a hero. You're already a hero because you know the system better than anyone. You can focus on other things. Maybe use your new-found credibility to build some consensus among your peers and managers. Then when you make your case to replace the app, it won't just be you.
This can be generalized: refactor the code to the point where what you need to next is easy to do.
I'm a fairly green developer (3 years professionally) working on an C# application that suffers from design schizophrenia as detailed in the article. I'm more or less the sole developer assigned to it now.
Although I am new, I feel very strongly about my 'pride as an engineer' in my application of sound design, knowledge and rigor to the best of my abilities. So it discourages me greatly when i hear a lot of "Uhh, that was a long time ago...", "We had tough deadlines..", "Well developer 'X' and me disagreed about that..." from the former maintainers of the code.
I've taken to a rash practice of just tearing out tightly coupled, untestable, duplicated and poorly designed code by the roots. I feel like i am somehow passing a 'holier-than-thou' judgment on the work of my predecessors. But due to it's tight coupling i can't have much confidence about even seemingly simple changes since they always seem to come back with a wagon of defects. The new code i write surely isn't perfect and has it's own defects that come back to me, but i feel much better about the code in many ways and i am always open to criticism and discussion over my design choices. If anyone would ever give them serious study...
I have this gut feeling that if i leave the code i touch messy that i'm not doing my job and the next poor soul to come behind me will lose weekends to trying to clean up an even deeper mess. I feel judgmental even voicing these thoughts.
That's not a terrible approach. Depending on the context the code is running in, that might be a worthwhile way to maintain it, especially if there's not all that much different between doing that and working on it as is. If it's taking you the same amount of time, why not leave it better than you found it? The important thing is that you finish the job, and it sounds like you're doing that. You're not making a three-year plan, not telling anyone about it, and doing a little bit every day. That way lies the lava layer.
Me personally, I have like four other projects I could be working on. Give me a choice between working on old legacy and working on new hotness, I'll band-aid the legacy every time and just get on with life. I didn't use to do it this way. I wanted to make my mark on the legacy codebase, prove that I was better than it. So I'd tell my boss I'd need the rest of the week to implement this and then dive in. I'm happy I don't need to do that anymore.
I think this statement sums up my attitude almost perfectly. I have a notion that if I channel that line of thinking into positive structural changes in the code then I should while I can. I just don't know whether or not it is a good attitude to have.
The amount of tests available on the previous implementation can go a long way towards lighting the way and bounding the level of required efforts. Code that is deeply embedded, untested and "ugly" is just a recipe for pain however.
My project is about 200KLOC C# 2.0; we have a single .cs file 13,000+ lines long and no, it's not generated. We have a single method 900+ lines long in our DAL containing 23 separate hand coded SQL queries and no, it is not a report. Our codebase contains exactly 14 unit tests. 10 of which were written by me. Your prayers are welcome.
Hello me, from a parallel universe!
You've basically described the project I'm currently assigned to and the situation I'm in as well, and I share your enthusiasm for "weeding" out obvious turds and warts from the codebase.
Unfortunately I've reached a new kind of roadblock when it comes to improving and refactoring the project: I've picked all the low-hanging fruit, and now if I tug at any of the remaining lose ends the whole thing will unravel.
Basically the remaining pain points are architectural ones (that will hurt us at scale) and that require attention but at the same time are large enough that management won't approve the necessary resources to work on them.
Honestly I almost feel as if I've fucked myself over by making those improvements: the project is now mediocre as a whole, but just good enough so that nobody really cares about improvements any more. Had I just let the ship sink, maybe something better would have come out of this whole mess in the end.
Not sure what the lesson here really is! :|
1000x times this. And making things coherent will make it a lot easier to put the new hotness in too. Add in the new hotness only when you can do it in a coherent way. If you can't do it in a coherent way, you might need to refactor until you can get there.
You might never get there in a legacy codebase, but if everyone took this attitude from the start, you might be less likely to end up there.
You can put new things in and change architecture, but you always have to do so coherently -- which does mean you can put in fewer new things, and sometimes not the new things you want to put in as soon as you want to put them in.
And oh, yes, so very much: "don't treat organizational problems as technical ones." Without understanding the organizational disfunction at the root of your technical issues (it's almost always there), you're doomed. Of course, then you've got to figure out what to do about the organizational problems (or how to operate succesfully in their shadow), which is not what you find the most fun part of your job, but that's how it is.
I've personally switched patterns over the years when adding features, always telling myself that I'll go back and refactor the earlier work.
Would be interesting to know if there is a relationship between years of experience in programming and how likely you are going to inject the latest "cool" design pattern / framework into code. Personally I have become far more cautious about this stuff. I'd rather adapt what is already present than subject anyone else to an abstract monstrosity I just cooked up after reading the latest hacker news post on the latest hype.
I've seen in with RPC calls (hand-coded HTTP -> Hessian -> Protobufs), serialization languages (XML -> Json -> Protobufs/Bson), ad nauseam.
I think this kind of changes is the natural state of multi-year development efforts. On a 10-15 years old code base, you'll see all sorts of things like that cropping up and it's incredibly hard to steer clear of these kinds of things.
What are the alternatives? Wholesale replacement of all uses? Wasteful in resources! Not introducing new technologies? Offends the sensibilities of the devs...
That makes them unable to solve the domain problems. They just follow the spec written by business analyst.
With no interesting problems to solve they get bored and look for "new" and "cool" technologies to make their work more palatable.
The easy answer of "regular rewrites every 5-7 years" doesn't sit well with paying customers or management.
Just as it's vital to accept that business requirements will change during a project, and structure your methodology to accommodate this, the same is true of technical considerations. And the same solution applies, viz. Agile: break up any "redesign" into small steps that can deliver concrete benefits inside a two-week period. Don't treat the "transitional architecture" as a temporary thing that doesn't need to be good, doesn't need documenting etc.; like with a big database migration, you make sure that at every stage it's something that you can develop under, and you know what you'd do if you had to abandon the plan at any stage.
If you try to keep a codebase on 2003 tech forever, you're storing up trouble for later; devs will become harder to find and more expensive to hire, and eventually you'll be forced into a Big Rewrite, and we know how those turn out. Like with deployment, better to embrace Continuous Re-Architecting. If it hurts, you're not doing it often enough.
> Look at Stack-Overflow ... they also use only static methods for performance, we should do the same ... Gordy had dismissed Ina’s love of unit testing. It’s hard to unit test code written mostly with static methods.
I also happen to like static methods quite a bit, precisely because I thought they were EASY to unit test. I thought that a static method, which is essentially just a function in the enclosing object's namespace, would be EASY to unit test. Input -> output. No state.
Am I digging my own code grave here? What am I missing?
public static doSomething() {
Database.write("Did something")
}
It can be a useful tool, but it causes all kinds of weirdness which can completely mitigate any advantages. On my current project I had to strip out most of our uses of PowerMock. While the tests pass on my local machine just fine, many of them fail for no good reason on the CI server.
Offtopic: Time for start using Docker? :p
Well that's why I suggested using Docker: To ensure both environments are exactly the same.
You say they are virtual machines, but how do you provision them? I assume you are using puppet/chef/salt/ansible and creating them from scratch, as opposed to uploading whatever image you used in dev, like you would with Docker.
Presumably where the definition of badly designed includes "uses static methods"? That's something that I hold true, and teach to other people as true, but I have started wondering about it. Whenever I ask why it's bad I get told "it makes testing hard". Well, Powermock can make testing it possible - "but powermock is only for badly designed code". What's badly designed code? And so it goes round.
Is it really that conceptually bad? Or is it just that our testing tools are insufficient, or that the language itself did not make provisions for testing all of its features? I've seen a lot of code recently that uses dependency injection purely for testing, and uses builders instead of constructors all over the place, and it feels like a lot of effort and complexity to avoid using PowerMock.
(Of course, Java was designed by people doing the best they could subject to particular constraints, and is now constrained by backwards compatibility. But other languages provide the same kind of features in better ways (or provide syntax that means builders and dependency injection are much less effort, depending on which way you look at it))
Pragmatically, good design is not the only concern; if you're not writing a library with binary-compatibility concerns, then IMO the syntactic overhead of builders is often a cost that outweighs the benefits. But the design costs of statics and constructors in Java are real enough, and there are very direct costs to using PowerMock (tests are harder to maintain and more brittle as they have to specify which classes they're rewriting, and it can conflict with other things that do bytecode manipulation e.g. test coverage tools).
public static doSomething(Database db){
db.write("did something"
}Basically, this concept means that it's static all the way down, from the top layer to the bottom layer.
In short: static methods are easy to test, methods using static methods (whether static or not) are more difficult to test.
My takeaway: the bar for moving functionality into non-static methods (which will still be decently easy to test) is quite low. If you call a static method, be prepared to own all its functionality when testing the calling method. Taken to logical conclusion, public static methods should be quite small.
1. First they get devices from one vendor. They set up ACLs in certain way: hardcoded IP addreses/ranges, hardcoded ports.
2. Then come devices from another vendor. This addes another layer.
3. Another vendor comes with an acl automation tool and tells team to create objects/references for ip addresses/ranges/ports. Some guys follow it; some thers don't.
4. You can see a big mess of things created by many vendors, many software automation tools, manual additions, etc.
One day, some guy changes the object/reference; all of sudden, production outage: add another layer of manual edits on the top.
That's why they preach "if it aint broke, dont fix it, but add another layer on top of it".
The constant narrative in the example is developers having no idea why and where things are built a certain way. With more information and verification, gradual changes and refactors can be more manageable.
Avoiding Lava layer applications is not feasible, so calling it an anti-pattern is silly.
Further, and I think this should be more and more clear to people, _any_ retained unencrypted data is _compromised_ data.
Unfortunately, our society is on the way to being completely dependent upon centralized systems and institutions which are unaware of their limitations, and in fact we live in an age where centralization is considered a virtue, when it is instead a significant liability.