The rise and fall of the dungeon master on software projects
medium.com
medium.com
If you think this new trendy solution is going to be cleaner or less hacky, and still cover the same (or more) use cases, you're rather naive.
80% of the ugliness comes from getting the last 20% of the functionality (maybe even 90/10). It is trivially easy to re-engineer a system that is 80% less ugly/hacky, and covers 80% of the functionality of its predecessor. Then everyone slaps themselves on the back because the refactor is going great!.
Not it won't. It will just prove that the rewrite was justified.
>Good developers turn into Dungeon Masters because this is what the system demands.
They really don't. Good developers realize that there is an endless of cycle of the old being replaced by the new and if you can't deal with that you are not suited to a career producing software.
The whole thing reads like the author encountered one or more people whose self esteem is so fragile that they are unsuited to the commercial software industry. It sounds like those individuals had a lot of trouble keeping their own (fragile) ego out of technical and business decisions.
>The moment a different implementation shows that a lot money could have been saved, then the cost of keeping the old software for too long is exposed too. And this can be scary: imagine you discover a wrong design decision you made 10 years ago costed millions to your company… would you happily expose it?
Yes. Exposing it presumably then saves the company millions going forward and any halfway competent boss will respect you for putting the organization first. Shelve your ego, it's called being a professional. Never mind being professional, its also a matter of acting like an adult.
The article isn't so much about dungeon masters as it is about people who shouldn't be programmers.
knieveltechs' Paradox...
The rest of our stack (the main backend API), which was poorly implemented by a non-proficient developer (which i wouldn't consider a bad developer, merely non-experienced; he eventually moved jobs), is currently done in NodeJS/Express/Sequelize and IMHO should be scrapped and rewritten.
Recently, during a conversation with a new hire frontend developer, he mentioned the ETL sub-project should be done in NodeJS as well, for same language consistency across the stack and i basically replied that if he was willing to rewrite it (with the same features i get from solid, highly featured mature community libraries) in NodeJS himself, he'd be free to do it :-)
Unfortunately, the CTO <rant>which IMHO is pretty null at managing people/software projects and sole aspiration seems to be becoming the CTO of the next Trello</rant>keeps throwing out buzzwords as "Kafka based event data driven application", "snappy websockets driven UI" and "microservices", while in the last year was hardly able to get a buggy over engineered MVP up and doesn't seem to have taken a Q, which means i'm looking for a new job. In the middle of all this NodeJS madness, i'm hopeing i have not become a dungeon Master myself :-)
I have bad news for you...
It seems that people use many different heuristics:
- We have an expert, surely his advice is good.
- We have an expert, surely he is bullshitting us for his own benefit.
- The subject matter is complicated, surely it must be useful, because the problem is complex.
- The subject matter is complicated, surely it's because people working on it over-engineered the solution.
- The thing is old, surely it has proven it's practical use.
- The thing is old, surely it uses obsolete technology.
- There are lot of features, surely the thing will solve my problem.
- There are lot of features, surely it's not all needed.
As above examples show, in absence of information, decision making will be subjective. And that's where the article falls flat, IMHO, because it wants to give an advice, but it can't, because there is probably no rational advice to be given, aside from, have somebody else (who you trust) understand the original system as much as possible and then tell you.
The article reminds me of a similar situation I have at work. I work on a large legacy application (but I am not a DM, I am not really invested in it yet), which should be slowly replaced by company, because we have a costly dependency on some vendor. Unfortunately, the new team doesn't want to consult our team!
Then I think you are a Dungeon Master anyway. Whether you like it or not. You'll be pushed into the role. Advice you think is essential will be ignored and it will be painful to watch a lot of the same mistakes being repeated.
I think you can end up in this anti-pattern in two ways - either the old maintainers don't want change or the new ones don't want to listen. The result is the same.
I am well aware that today's society prefers foxes (people who have shallow knowledge of many things) to hedgehogs (people who have deep knowledge of few things), and I think it's wrong and unfortunate (because it actually quite heavily depends on hedgehogs). The article is, IMHO, a good example of this line of thought.
Yes, the magical silver bullet, that kills the dungeon master! Snake oil extracted from within the great depths of brogramming. Acme Elbow Grease to help develop scalable™ MVP's with possibilities of future growth. That's it - 42 is outdated and irrelevant anymore, the answer now is Node.js
Yes. Obviously. That's my job. Are you saying you wouldn't?
The whole article has this really weird assumption that people will naturally be ashamed that work they did years ago with a more limited budget/team/understanding of the problem is now showing cracks. But that is the natural state of software development. It happens to everyone, everywhere. A developer who can't accept that has a serious ego problem, and a manager who can't accept that should not be managing developers. Either way, the problem is the individual, not the design pattern.
I would recommend reading this article by Chris Grobmeier: https://www.grobmeier.de/the-10-rules-of-a-zen-programmer-03...
The philosphy in it is a good counter to most of the points mentioned in the post.
Assigning malice to the people who understand how to work within the technical debt on a project is age-old programming nonsense. It was - and continues to be - a stupid thing to do.
This article just makes it a bit more conspiratorial, and re-writes the old story to become less falsifiable. Rather than accuse someone of simply being counterproductive to protect their job security, the fault is found in their ego and subconscious actions.
It's comforting to find a way to frame a problem such that you can blame a person for holding out. But the barrier to entry on a codebase is just another form of technical debt. It's no fun to accept that codebases evolves in such a way to accumulated debt, or that the development plan might have even accelerated that accumulation for other short term goals. It's doubly no fun to think about scheduling developers to pay down that debt.
I thought the author acknowledged that it's complicated and that it has many aspects to it and that it's not easy to solve at all.
> useless drivel on medium
You are entitled to your opinion. I personally feel it was a good read and I share his sentiments of frustration and resent
In general, I think this is the issue. Dev "experience" is acquired knowledge that is no shared/communicated.
> The dark secret of the Dungeon Master is that he knows every trap in the existing legacy software, because he was the one to leave the traps around.
It could be that your Dungeon Master isn't responsible for the traps; perhaps they were handed to him from his predecessor, or they were the result of a bad tradeoff — in my experience, usually quality being sacrificed in favor of short term time and cost (and much to the detriment of long term development), time and again, without ever allowing the dev team to revisit the accumulated technical debt, until it becomes so back breaking that we throw our hands up and — against what seems to be the prevailing wisdom — cry for a rewrite. If you start from such an accusatory stance as the above, well, yes, you and your "Dungeon Master" are going to have an adversarial relationship.
I feel like when I've been in this position, it's not the code that I'm trying to protect: the code is awful, and I'll readily admit that. However, it meets a set of requirements, albeit badly; but more often than not I see the rewrite dropping crucial requirements in favor of "simplicity", or "we're going to start with a minimum viable product" — after which you'll repeat the history of the same set of mistakes that the original made by allowing your product to grow organically, all the while ignoring the existing implementation that is ripe with knowledge of how well that won't go.
> The worst domain expert is the one whose expertise is built from the intricacies of the existing systems
I agree! And yet this article seems to seek to vilify this person. We shouldn't have domain experts whose domain is the cruft of the system, but as long as our software development processes are so broken that the problems with the current system can't be addressed, I'd much rather have someone who is aware of what the limitations of that system are, what corners can be cut and what the risks of cutting them are.
Perhaps you've ran into someone who is really as counterproductive as the article makes a Dungeon Master out to me, but I've never so feared for my job security, which seems to be the core reason provided for the Dungeon Master's behavior and position. The article comes across as incredibly deadset on a us vs. them mindset, to the point where I question whether the author should take a step back as ask if he isn't himself part of the problem.
> They are going to find themselves less productive than their younger peers, good in useless things, but embarrassingly slow in what matters today.
This struck me as incredibly ageist when I read it. He recants somewhat at the end,
> One thing that I didn’t intend was to raise a generational/age issue. Thanks to Ann Witbrock for pointing me to that. It’s not age: it’s attitude, systems and vicious circles.
I have and it's very scary indeed. I've heard "You will be fired if you don't support technology X" simply because that's the technology the DM used originally and is more familiar with.
I'm not disagreeing with your other comments, but I think the authors main point boils down to when a DM's ego becomes tied to a system in need of replacing, the act of replacing said system can feel like a personal attack to that DM. If that developer is in a or has risen to a position of power it can feel even more intimidating to suggest other technologies.
That being said, I've learned to expect such behavior. It's part of the job. A great skill to have is negotiating and communicating why the system needs to be replaced in such a way that the DM is collaborating and seeing the value the new system can produce. I'm going to work with people everyday that don't see or don't like how I'm trying to solve something and their concerns need to be addressed for us to continue working together.
> The dark secret of the Dungeon Master is that he knows every trap in the existing legacy software, because he was the one to leave the traps around.
I take issue with this quote as well. In my experience, I know about the traps because I've seen what happens when they get triggered and had to recover from them - often at great time and expense. Put another way traps are edge cases and like any system it's easy to account for 90% of the complexity it's the remaining 10% of edge cases that cause the headaches.
When I hear a young developer say something like "We’ll just use a const and assume each day is 24 hours long because it simplifies the code immensely".
When I then point out they’ve stumbled into a trap because their program will now fail catastrophically twice a year (daylight savings, when you have a 23 and 25 hour day). I’m not doing it to be a curmudgeonly dungeon master trying to protect my turf. I’m pointing it out because I’ve dealt with that pain when I got bitten by DST in the app I maintain. I was young once too.
It’s easy for some junior to say “This error will rarely occur it is not worth introducing complexity to handle it.”
They aren't the ones who will get called out to fix it when the 'unlikely error' inevitably crops up. In the case of DST twice a year for the 15+ year life of the app adds up.
Edit: I'm sure someone will mention UTC - I agree better to use this if underlying DB supports it.
In other words, while the rewrite is common, having the original author around is uncommon.
And I've never seen this approach NOT working. "Working" meaning a system that you can reliably change.
I'm sure projects fail for many, many reasons, and inertia plus ego from an original developer that's now powerful is certainly one of them.
On the other hand, the number of half-baked ORM replacements for SQL I've seen represents a staggering amount of wasted labor, labor that could have launched a number of great tech teams if they'd had the wisdom to learn from someone who was coding while they were in diapers.
I think I find the whole thing so offensive because node plus microservices are put out in the essay as the thing that a hot new dev knows better than her elders.
So -- computing models are cyclical; they wax and wane for social and practical reasons, and admittedly, beyond all predictions javascript has been massaged into an amazingly fast thing given where it started.
But, really? microservices as an approach that's just too hard for some old fogey? The same fogey that probably could concatenate piped unix commands to get real shit done in the '80s, and had a solid understanding of awk syntax?
Annoying.
On the one hand, I strongly agree with the sarcasm of this: "Because...you know...SOA and N-tier systems didn't exist before Node. "
That is well said.
On the other hand, something must explain the continued rise of Node and Javascript.
Since 2010 I've been a huge fan and promoter of Clojure. I keep thinking it is going to sweep the startup world. And that keeps failing to happen. And I keep wondering why.
Last year I worked at a startup that was in the startup incubator run by NYU in New York. The incubator is an office at 137 Varick street, and there are something like 20 to 30 startups up in there (the number varies a lot as students of NYU are allowed to use the space for short periods).
One thing that surprised me was how completely Node had gained the support of the newer startups. It was definitely the hot new thing. Some of the older startups were using Rails, the rest were using Node. And Node very much has the fire that Rails had 10 years ago.
My startup used a mix of Clojure and Java but we were a rare thing (I wrote about this elsewhere). None of the startups were using Groovy or Java or Scala or Clojure -- the vast resources of the JVM were largely ignored. Nor do I recall any of the startups using C#.
Why is that? I am not sure, but I can make a few guesses.
You mention SOA. The culture that grew up around industry standards such as WSDL lead James Lewis and Martin Fowler to complain about “a complexity that is, frankly, breathtaking“:
"Certainly, many of the techniques in use in the microservice community have grown from the experiences of developers integrating services in large organisations. The Tolerant Reader pattern is an example of this. Efforts to use the web have contributed, using simple protocols is another approach derived from these experiences — a reaction away from central standards that have reached a complexity that is, frankly, breathtaking. (Any time you need an ontology to manage your ontologies you know you are in deep trouble.)"
They then linked to this to make their point:
http://wiki.apache.org/ws/WebServiceSpecifications
Arguably, Node offers an easy approach to "SOA lite". Consider how forbidding the previous link is, and then consider how friendly an approach is offered by a company such as LSQ (which started off specializing in Node):
Especially since LSQ is about to roll out an automated processes for microservices on a stack to discover one another (which app runs on which port), a relative novice can get much of the benefits of SOA, quick and easy.
Also, of course, Javascript can be used on the frontend and backend, and nowadays there are a lot of programmers who start off as frontenders and then later learn the backend, and want to bring their favorite language with them.
I could counter-argue that Clojure has Clojurescript and Om-Next, both of which are very, very cool, but I must admit that the JVM has no on-ramp as easy as what Node offers (and programmers who use Clojurescript, but not Clojure, have been rare, at least till very recently).
Also, I've wrestled with enough too complex build scripts that I now appreciate how simple "npm install" is.
So maybe some programmers, who are still in the early phases of their career, are over enthusiastic about Node/Javascript. We should all still admit that what Node achieved in terms of ease of use is fairly impressive.
Node has its uses for prototyping and quick projects, at least for me.
I honestly spend about 1/3 of my time in the Java ecosystem -- it has its uses, I've done a lot of it (dev and operations), but I personally dislike the general weight of dealing with server side Java - app servers, etc.
Language and ecosystem, for me, are dictated by the needs of the particular project. I said 1/3 Java above, the other 2/3 for me (right now) are C and Node... Across 4 projects.
I guess if you think back, JSON has the same story that the node story you tell has; it was easy to work with for developers and didn't require a lot of theory or tooling. That makes sense to me as a great pattern to get adoption.
The much larger point though is that we should be able to talk over pluses and minuses in different ecosystems without calling somebody outdated or too old or too slow.
As an aside, don't confuse Java EE with J2EE - the new enterprise Java system is a far simpler development environment than the old (I almost gave up on enterprise Java myself before this transition).
SOA COM Corba and so on and so on...
the only thing that keeps evolving is the buzzwords
I guess there is more than one anti-pattern