Dark Matter Developers: The Unseen 99% (2012)
hanselman.com
hanselman.com
There is a lot of legacy code that is still working fine. There is a lot of old C# code bases that are working exactly as intended in now unsupported .NET versions.
One of the constant annoyances of some of the newer tech stacks is that developers constantly break the API meaning that when you upgrade the framework version you have to fix code and probably tests as well. It is absolutely infuriating when an API is changed when the old API was working perfectly well.
Good thing Google is reading this.
Sorry guys but if you can't manage the task of providing randomness as a stable basic lib then maybe all you have is an experimental language not fit for production? It's scary how the security and correctness of the lang doesn't match the namespace problem that exists currently.
The single owner of one of the most popular crypto libraries hasnt responded in years to anyone. Hundreds of downloads a day still. Evangelists please take a step back and think about that.
The newer .NET Core most stuff is now compatible and it is pretty easy to write code that is compatible. Stuff that isn't compatible normally needs you to install a compat shim that Microsoft provides, but I've not needed them so far.
My major frustration is with the JS frameworks. I wrote some perfectly good Angular 4 code and I had to change quite a lot of it to work with newer versions of the framework.
Naturally I try to learn about everything that pops up, languages, trends, whatever.
But is a peace of mind of staying with such platform languages, while watching others coming and going, really feels gratifying to me.
I don't blog. I don't go to conferences. I don't go to hackathons. I don't have any meaningful presence on GitHub. Hacker News is about the only place I have even the slightest footprint, and most of what I talk about here has nothing to do with my career as a software engineer.
So what? The implication in this article is that if you're not out there trying to be visible, you must be some crusty old COBOL developer locked in the basement.
In my experience, the people who are the loudest are rarely the people who are doing the most. If you spend all your time trying to prove to strangers that you're hip to the latest trends, just how much time are you spending doing anything of actual value? Not everyone who has a big public presence is actually doing anything. Most of the discussions online and presentations at conferences are nothing new. Most of it is self-promotion. When I see people get all excited about tech, it's usually because it's new to them, not because it's a new idea.
This is something I've wondered about a lot of social media users to be honest. I mean, if you're seeming spending all day on Twitter or what not, how are you finding the time to actually do any work? Do any of these super outgoing social media addicted conference every week developers actually have a full time job? Or is their entire career about promoting themselves?
I would absolutely hate it to be interrupted in the middle of a good coding session by someone who wants to talk to me on twitter about the hype of the day.
Most of these people love technology. They love learning about technology. They'd be perfectly happy working in technology, but because monetary success ends at being specialist consultant in a super lucrative field.. They stop.
Now they work in strategy and marketing around technology. Why? Because it's more lucrative. The job is more fun. They get to meet really, really smart people without having to work 60 hour weeks for $250K a year. (Which is great, of course, but these people have the soft-skills to add a multiplier to that amount.)
I'd be happy working 9-5 as a CRUD software developer with a great team in an OK company.
I'd be happy working in a very specialist field, earning $250K with an amazing team, in an amazing (funded) company (for a few years before shareholders start making demands).
I'm more happy working random hours around the world, meeting and talking with the smartest people in technology at a range of companies. I get to learn about what they've achieved, how they've achieved it.. Without having to burn the candle at both ends.. And I get to just play with whatever tech I feel like playing with.
They're different careers. A lot of people dabbling, or dipping their toes with blogging etc. are just doing it for a better portfolio when they company hop.
But we can now scroll forever. Slowly and jerkily.
On the language front, we really should have had a replacement for C/C++ by now. We don't. D never got traction. Go is mostly for web back ends, where it has really solid libraries heavily used within Google. Rust looked promising but was taken over by the "functional" crowd and is doomed to Haskell obscurity. C++ itself is now trying to fake ownership tracking at run time, with mediocre results. The abstractions always leak because too many old things need C pointers. Java was supposed to replace C/C++ but became tangled in its own libraries and ended up replacing COBOL.
There's been some progress. There's now grudging agreement that the way to go is type declarations with some type inference. Having to type everything is too much of a pain. Not typing anything makes for unreadable programs. Function definitions really do need type info. So C++ added "auto" and Python added "advisory typing", moving both towards the center. Which is about where Go started.
Both true, yet engineers keep switching to newer frameworks and libraries anyway, rather than sticking with what they have and just building stuff.
The question then becomes, assuming they are rational, why?
I'll propose at least one major reason: efficiency. What's improving isn't what is possible to build with these technologies, but how effectively those things can be built, to a reasonable degree of reliability, in the same amount of time, even accounting for the greater difficulty in learning and using the technologies.
That's where progress is being made here. Not in the what but in the how easily, how quickly, how well. You could do anything you can do in React (say) with vanilla JS, imperatively updating the DOM with cumbersome APIs etc. The question is how much longer would it take / what more could you have built in that time using React.
I think that's unfair and inaccurate. Rust is going to remain niche because most developers are happy with garbage collected languages and have no need to switch to Rust. Rust will eat a bit into C++'s domain, but I don't think it'll ever take over the world. GC is good enough, works great now and worked great 20 years ago.
Garbage collection is great until then.
What makes him think we don't have a balance already? Most devs are the quiet ones who just do their job and then go home. Aside from participation on HN, I consider myself to be be one of those quiet ones. I work on business problems with whatever tools are appropriate, which includes both modern stacks and legacy codebases.
I'm not sure what other balance he is seeking.
So, I think I am pretty well balanced.
1. Most of the new tools are pure nightmare to maintain. RoR has breaking API changes often enough that you need multiple people to maintain a large codebase, whereas in PHP you'd only need one person because things don't magically break that often. And MongoDB is so convoluted and complicated that almost nobody can deploy it correctly. And then there's "new" container techniques, which are basically just an admission that you have no f*ing clue on how to make your new tool set secure. In most cases, the loud new technology is just over-funded hipster stuff, but not ready for the real world yet.
2. Posting things online (e.g. Source Code) will not only attract the attention of peers. You might also get flooded with rather demanding emails by technically incompetent people who complain that your free source code release didn't solve their problem.
Also, I wonder if the author is aware of IRCs ongoing popularity for private development discussion rooms. It seems many people thought Slack was new, but some of us programmers have been hanging out like that since 95.
It has nothing to do with the code being spaghetti or not. It's about a level of abstraction where I no longer consider something made with wordpress as something created with PHP. The PHP was allready created and then someone just clicked a few menus to choose what existing PHP he or she wanted.
The more fitting category would be just calling such websites "Wordpress" websites and not PHP websites.
I can deploy a python web application I have written to a new web server within minutes by clicking deploy in bitbucket.
So using your definition:
* The code is already written.
* I clicked a few menus and I had the python code I wanted running.
It isn't a python application. However the web application when it receives a request will run python code. It is clearly a python based web application.
So if I got someone else to press the deploy button in bitbucket, it suddenly isn't python code? That doesn't make sense does it?
> Why don't you consider each and every website running PHP also a C website?
The web site/app logic is written in PHP, not C. The runtime for the vast majority of PHP deployments happens to be written in C.
Additionally while I did say interpreted earlier that really isn't true anymore. PHP runtimes these days tend to break the PHP script down into byte-code. There are alternative runtimes for PHP just as there is Java or .NET.
There are at least 4 I can think of. Two of those aren't C (C++ and .NET). However there is nothing stopping you from writing a runtime that runs on the JVM / Lisp / Go / Brainfuck / Lol code / 68k assembler.
You completely ignored the most important sentence in my comment which addresses this.
> The web site/app logic is written in PHP, not C. The runtime for the vast majority of PHP deployments happens to be written in C.
The logic is written in PHP just as much as it is written in C. The logic is actually written in the language the creator used. Which is the wordpress UI.
I did not. I've addressed it at least 2 times now. If you choose to ignore those responses that invalidate your point that is of your own doing.
> The logic is written in PHP just as much as it is written in C.
Nope. I have already shown this assertion of yours to be false as there is no requirement for the runtime to be written in C.
> The logic is actually written in the language the creator used. Which is the wordpress UI.
Issuing commands to a program (which is what is happening) does not make those commands a programming language.
So yeah, PHP is still a thing.
Eh, that's not really the case anymore either. It's made many improvements over the years. Not that I'm saying it's some wonderful language, but you can't really compare it to the PHP 4/early 5 days. I think credit is deserved to the folks who have brought PHP to where it is today.
On the other hand, not even using a service like Heroku can save us from constantly having to patch or update the Rails apps.
This. Happens so frequently, that making things OSS now carries an additional burden. I personally find it very painful to e.g. ignore the PR or issue opened by some hapless person in a bad situation trying to figure out the most dumbest thing. It makes the alternative (never OSS'ing anything) sound better, if only for my own peace of mind.
https://guides.rubyonrails.org/maintenance_policy.html
> RoR has breaking API changes often enough ... magically break that often
According to the support policy, nothing magic about those breaking changes: "Breaking changes are paired with deprecation notices in the previous minor or major release."
That said, I was referring more to the Ruby on Rails ecosystem in general. Our production deployment has 80 gems, which I'd say is a normal number for a big Rails webapp, and sadly many of them do not follow semantic versioning, so that minor gem version updates might contain unexpected breaking changes for us.
The advantage that old languages like PHP or C++ have here is that their standard libraries have been around for so long that the API rarely changes these days.
Once I learned to listen to incompetent users, I started to create usable software.
My gut reaction is still, "RTFM you're doing it wrong." But now I follow that up by figuring out how to make the software work like the user thought it should or I change it so they never get in the frustrating situation in the first place.
What I finally realized was that if one person got frustrated enough with my software that they took the trouble to post an incoherent and technically incorrect rant on Github then it's likely that 100s more users had similar problems and said nothing.
Now I receive effusive emails about my Open-Source almost weekly.
But after reading the article, I thought that the 99% number was about developers not joining the discussion with other developers. And for that purpose, I'm not sure if open source is still a good idea.
An example of this is net core vs net fx. Net core broke many things, has not implemented full compatibility from day one, introduced netstandard , which as of latest version is not implemented by netfx, introduced new toolset (dotnet) aside msbuild (bear in mind that VS still uses Ms uild) and so on.
This is all a matter of incentives both for tech companies and for their workers - they are often rewarded for short term/easy to see by managers impact, not for bugfixing, stability or long term support. It was not always like this - in the 90s MS would be picked because they took care to not break stuff. Migrating .Net 1.1 to 2.0 was super easy.
While there have been legitimate problems with this process - for example, the explicit ‘go live’ given by Microsoft before things were truly stable, there was a choice made to develop in public and the net result (sorry) seems positive to me.
I think you've stumbled on the core issue here. For an unchanging company for which their digital business does not need to evolve very much, having a basic IT department may just work. But as we digitize more and make technology a core competency and critical differentiator, a legacy, inflexible tech stack becomes a burden. It doesn't let you add the features that your customers may want, features which might be critical to a continuing and successful business relationship. Your customers then have similar demands from _their_ customers... all the way back to the primary sector.
Antipattern one is the one where you just get on the treadmill and churn, baby, churn. Constantly replacing things, spending way too much resources on tech and not enough on meeting customer needs. This is sort of the "new sports car every two years" model, or as a certain person who hates this site enough to make linking to him pathological calls it, the Cascade Of Attention Deficit Teenagers model. Nothing stays around long enough to enter long-term maintenance, because you keep chasing after the hot shiny thing instead of having the discipline to maintain a mature product even though maintenance isn't fun.
Antipattern two is where you don't have the resources to do maintenance or to chase after the shiny thing. This is basically treating the code as a monument, or as a museum exhibition -- every time a new feature is done, you hang it in the museum, and then you go work on the next feature. The problem with this approach is that Someday The Bill Comes Due. A dependency that you have hits EOL and nobody's made a new version in three years. You spend your time writing code to duplicate features that your language's standard library added four years ago but that you don't have access to because you haven't updated your stack in eight years. New features get harder and harder to add because it gets harder to reason about what any one piece of code in the program actually does, or everything is bottlenecked by the one guy who understands the codebase. Your bus factor is one if you're lucky and zero if the bus came.
The pattern is treating your code like a house. It needs upkeep. If you just leave a leaky bit of plumbing long enough you're going to have a flooded basement, so you bite the bullet and you fix the plumbing when something happens. But you don't trade in for a new house every two years either. You have a project list and you tackle it as best you can and sometimes you take a few days off work for a big project like painting or what have you. And you build equity that lasts.
Some people maybe don't need help, guidance, or something like that, maybe just maybe we like the things as they are now.
Maybe we'll start doing all of social stuff some day.
It's just okay.
If you or others such as yourself did blog about success stories, or real-world experiences in General, other engineers would perhaps have a more sober perspective of the bleeding edge and would make more informed decisions when deciding to switch stacks instead of just "hey its cool and the latest fad".
Again, I don't really expect you to change your behavior at all, this is just a description of what happens in aggregate.
There are stories which could certainly help engineers around the globe, but those friends of mine just doesn't feel any emerging feeling to write about it.
I've been in some projects where more experienced team members very clearly reminded everyone that technology X would not be useful here, but others on the team just resented them for disallowing the cool stuff.
So this could also be a conflict between the company's goals (stable product) and the individuals' goals (cool CV).
TL;DR, I know this is an old article, but it seems there's still some perception that us dark matter folks are all just passionless drones in it for the paycheck, and while there's surely some percentage of us for which that's true, I think it's completely wrong-headed to assume that's how we all are.
Of course that's not to say being in it for the paycheck is necessarily a bad thing. Not everyone needs to be 'passionate' about their job to do well in it, and if say, my doctor or surgeon is good at what they do, I don't really care if they spend all their free time talking medical issues on social media or posting on Hospital News or what not.
I think there's just a lot of black and white thinking going on around this stuff, I.e. you're either this or that (I.e. cutting edge rock star or old crusty dinosaur...) and I just don't believe that's how it is. It's multidimensional and we're all on the various spectrums somewhere and we should stop putting value judgements on which side of the spectrum is better or worse. We all do very different things with very different requirements, so stop prescribing what's best for everyone.
Chris Coyier's The Great Divide (https://css-tricks.com/the-great-divide/), although nominally about front end development, does a pretty good job explaining this problem, and I don't think this is unique to FE at all.
The group of people he is describing I think largely don't bother to come to communities like this. They do their job, go home and only know about developments in tech if it is relevant to their job and most likely because it comes through in-house channels.
I know a lot of people like this. Many of them aren't even aware that automated testing is a thing. They hardly know anything about any programming languages or technologies except for those they work with.
That doesn't mean it's easy though. I still hate it and have to make a conscious effort to market myself.
That is, I think "dark matter" developers read blogs but they just don't write them.
As for conferences, these are widely seen as a scam. I remember sitting through a presentation at the end of a conference by the platinum sponsor where I sat next to some tired defense contractors from the East Coast and then drinking in the hotel bar later with the director of marketing for the sponsor who thought that he hadn't gotten what he paid for.
Scott could be a much bigger part of the solution than he chooses to be.
For sure, I see him doing what you say, but I also see him regularly retweeting and responding to comments and questions from accounts with few followers, and particularly so from people from minorities within the developer community.
There are plenty of examples of people I can think of who behave in exactly the way you suggest, mainly spinning the flywheel of boosting the online presence of already famous people and not engaging much with outsiders, but Scott not only isn't one of them but perhaps one of the best examples I can think of of actively doing the opposite of that.
onion2k I ask you to look again at his Twitter stream and podcast, and re-assess your evaluation of him.
Usually they control the conference circuit by donations and are able to de-platform people who would tell the truth.
It always pays off to look at the git history, which few people do.
It's not his job/role to give people breaks. He's not a Hollywood impresario, he's a blogger. If people are good, they will be discovered.
This is not a problem that needs to be solved, it's just something to be aware of.
You should listen to Scott's podcast: he has a wide array of guests, in many diverse topics (even topics that could be seen at odds with his employer, Microsoft). Most of them are people I've never heard of. If there are gatekeepers, Scott isn't one of them.
And yes, they learn a lot from me. Sprinkling some new tools like git and Docker, encouraging people migrating their systems off PHP4, etc. can't hurt.
Any migration could hurt.
I thought I was a “Dark Matter Dev” until this. But then it’s weekend so maybe I still am.
There's enough dysfunction to go around that all types of programmer can be uniquely afflicted, and usually are.