The trouble is that "don't build shit software" isn't actionable advice on how to avoid building shit software. For someone seeking to be a more skillful web software engineer, there's no information there.
You talk to one software engineer, and they'll say, "that's built in PHP, so it sucks; it should have been built with npm, obviously." Then you talk to another and they say, "I can't believe they built that with npm, the package management system is just terrible and the whole mess is bloated and there have been a ton of security issues. They should have used a static generator instead." And on, and on, and on.
Now compare this to a mature industry with real engineers who are able to say things like, "sure, that material might not be the most durable on the market, but unobtainium would have added $35 million to the project cost and ongoing maintenance costs would have gone up by another $5 million / year."
That right there is a comparison that developers can never, ever offer anyone that's writing the checks for a project. Nobody's bothered to do that work. We can't even accurately estimate the cost of a complex project given our choice of tools, let alone talk about costs and ROI with different toolchains for the same projects.
This is absolutely shameful IMO, and the reason why I cringe a little inside whenever I see the words "software engineer".
Also, those mature industries routinely overrun by millions of dollars and months or years, event though they are dealing with a fraction of the complexity and the costs and techniques are well known.
Software is the only industry where you are asked to build systems that have never been built before but that are more complex than anything ever built in the history of "mature industries", on top of astronomically complex systems, and estimate it with an accuracy of 5 to 10 days and a few thousand dollars.
What they do to the end user might be different, how the pieces come together, it is always the same, unless one is doing curriculum driven development.
Forget everything about responsiveness or whatever (which I find way more valuable than the tradeoffs one makes)--the lack of tight coupling is the real win of React and similar. I mean, if you'd like you can pretty easily build a server-side React app with NextJS and still get a lot of these benefits while using React purely as a pretty-dang-good templating language.
Old school server side rendered apps are underrated and its perfectly possible to make something that looks and behaves as well as a SPA app for most things.
Even in the case where you do have a specific frontend, I contend that there's still value in the separation because it very strongly encourages (coupled with tools like, shameless plug, nestjs-data-sec[0]) the writing of very clear viewmodels where thought must be put into what exactly should be exposed over the wire in a way that commingling database objects and template logic doesn't allow.
I think server-side rendered pages are properly rated at this point, not underrated; there are places to use them but they're shrinking as other tooling get better. As an example, even I write React almost exclusively because I write React faster and with fewer errors than any templating language I see in common use except maybe Razor (and Razor implies buying into ASP.NET, which, no). React/TypeScript are hard to get wrong and easy to burn through quickly. And I would call a React page rendered through NextJS a "server-side rendered app"--but I get the feeling you would not. ;)
You are seriously overestimating complexity and novelty of most software development, while underestimating engineering in other industries. I work currently as software developer and previously in a branch of a civil engineering.
Do you seriously think that other engineering disciplines to not have those small little problems that plague us all the time in software development? They do, but they can offer their solutions based on data, not just whim. They do also have codified standards, and guidelines and certification exams.
On the other hand software development is relatively new discipline - other fields had hundred of years to mature.
EDIT: and they are catching up with rapid prototyping - it is getting cheaper and faster.
I have a feeling that some SpaceX engineers would have something to say about this after one of their rockets lands on a floating barge.
That is because you are ignoring/left out the critical last part of the phrase you quoted: "and estimate it with an accuracy of 5 to 10 days and a few thousand dollars."
This is particularly relevant because this is a direct response to the following:
> We can't even accurately estimate the cost of a complex project given our choice of tools, let alone talk about costs and ROI with different toolchains for the same projects.
Anyone engineering hypersonic missiles would easily dispute that.[1] Or submarine-capable quantum communication systems.[2] [1]https://www.nytimes.com/2019/06/19/magazine/hypersonic-missi... [2]https://www.realcleardefense.com/2017/11/04/quantum_communic...
[1]https://www.scmp.com/news/world/russia-central-asia/article/...
But maybe its hard to admit it, lower down your ego and be a bit humble. Yes, software development is a creative work, but so are all other engineering branches. And yes, we have massive issues with code, process and people quality across whole business.
> That right there is a comparison that developers can never, ever offer anyone that's writing the checks for a project.
Sure they can. At Google there was an internal cheat sheet of "numbers every engineer should know" which included the cost in dollar terms of trading off engineer time vs. memory vs. CPU vs. network vs. user-perceived latency etc. (Actually there were two of them - the more famous one is the Jeff Dean version that traded off CPU cycles vs. L1-L3 cache misses vs. RAM latency vs. sending data over the network vs. compression vs. disk latency etc, measured in nanoseconds) If you had any moderately complex new system, you'd prototype a few different approaches on a small set of sampled data, do some multiplication to figure out how much it'll actually cost in production, then compare that against the engineer time needed for various optimizations. If you were in a revenue-generating area like Search, somebody (usually at the executive level) would also be calculating how much the project was likely to make or lose for the company and ensuring that's acceptable. Not all that unlike any traditional engineering field.
The thing is that the overhead for this process really only makes sense at Google scale. For the vast majority of new software projects, revenues on success >>> developer time >>> hardware cost. So you just assume that if you can get the job done in a way that satisfies the customer, it'll be wildly profitable, and the primary factor to optimize for is developer time. And if the developer can build it, the hardware costs needed to run it will be negligible, so just buy more hardware.
This is an artifact of the software industry essentially being a gold rush, where it's rapidly eating the rest of the world and replacing non-computerized processes that are millions of times slower. When you have 6 orders of magnitude to play with, you can play fast and loose with estimates, because any acceptable solution at all will make you a lot of money. When software has eaten the rest of the world and new systems are competing with existing computerized systems, then it's worth it to start paying attention to things like efficiency, security, TCO. There's some evidence we're approaching that point, but we're not there yet, and it's still more advisable for an entrepreneur to tackle an old-school non-computerized industry than to try and improve on existing software solutions.
Could someone alert Google's web developer teams (in particular the GMail frontend team) about that cheat sheet? If released products are considered as evidence, it doesn't seem like this cheat sheet is used.
[0] https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...
As well as having free-form pointers, C/C++ also have many, many dark corners.
What proportion of C++ programmers know that comparison = (c == c++); causes undefined behaviour?
> The thing is that the overhead for this process really only makes sense at Google scale.
The software industry is far larger than Google. If we want to claim that software is a discipline of engineering, we all need to have access to that kind of reference data, rather than trying to feel our way around in the dark in comment threads about blog posts about somebody's personal experiences.
I won't go so far as to say there's no such thing as software engineering -- a number of people working for companies with large budgets have well earned that title, I'm sure -- but the extent to which the term is traded like currency in our industry is disrespectful to the discipline.
> So you just assume that if you can get the job done in a way that satisfies the customer, it'll be wildly profitable, and the primary factor to optimize for is developer time.
The fetishization of developer time is one of the problems slowing the maturation of our industry. It ignores end-user time, it ignores management time; it often focuses solely on the initial development time of a project and ignores entirely the ongoing maintenance time for a project.
How many codebases are there out there that are gigantic balls of mud? How much developer time has been wasted on trying to maintain or refactor those codebases? How many really dumb security issues have negatively impacted companies or end users? How many comments has tptacek left on one little forum in one musty corner of the internet over the years trying to explain cryptographic and security-related best practices to an infinite legion of under-trained developers?
How many people have lost their lives to software defects? Not even the really tricky CS stuff, I mean the kind of defects that any other engineering discipline uses as case studies in their introductory courses.
Which case study would you choose for our Galloping Gertie?
Certainly software has done an incalculable amount of good in the world, and its impact is far and away net positive.
But while software has been busy "eating the world", it has rarely paused to also consider the cost and damage it has wrought and whether there are practices that could be standardized to reduce those negative effects. So I disagree wholeheartedly that we're not yet at the point that these things are important. Depending on who you ask, computer programming is between 60 and 180 years old. That's well long enough for us to begin asking questions like, "what data do we have to support this approach or that approach?"
There is still plenty of room for experimentation and research. There is still plenty of room for hobbyists and tinkerers. This isn't an exclusive-or thing.
As much as I really do appreciate blog posts like this one, and the discussion they tend to generate, and especially that there's a new one to read every single day (he said, wryly), I can't help but see every one of them as another indicator of the immaturity of our profession.
Part of this is because many systems are evolving, because business needs have human inputs and are not always "logical." At the edges where this human interaction happens, we expand the domain of programming to that of human psychology. There definitely are empirical claims we can make human-computer interfaces, but it begins to get a bit messy.
To be clear, I am personally interested in becoming more rigorous about empirically justifying claims I make about the code I write. But I’m also interested in code because many of the problems it attempts to solve map the very complexity of the real world. In many ways, abstraction is a properly philosophical exercise, and I think it’s impossible to separate a pure engineering discipline from this broader context.
Programming is multi-disciplinary, and I think that’s a strength.
We also have a multiplicity of building materials, tools, fabrication processes, etc. What's the standard way to make a steel object? Do you cast, forge, stamp, mill, sinter, roll, weld? I would expect engineering as a discipline to have more solid arguments for choosing which (combination) of those processes to use for a particular task than what is typically thrown around in discussions about choosing a programming language or paradigm.
The problem is that in many projects, especially small web-dev or simple automation projects, you simply don't know if they're going to be profitable. The margins are huge, since the upfront development cost is almost constant, you might lose $50,000, or you might make $500,000, or anything in-between, but you just don't really know beforehand. So obviously in this situation you want to become at least somewhat profitable as fast as possible, and then think about "doing things right" later. Is this horrible practice? Sure. But it's also the most logical thing to do here, because "doing things right" at the start is simply a huge risk that only pays off if you're successful anyway. And people would rather minimize losses than maximize gains.
On the other hand more traditional engineering tasks tend to require big up-front investments anyway, and "failing is not an option". I mean it really kind of isn't, if you need more money and/or more time you'll get that much more often here because the initial investment was so big already. So doing things right from the start makes complete sense.
Would love to see that. For companies determining what's worth what, this would be great to see, even just for the format.
1 engineer-hour =
W ms/search of latency
X GB-year of RAM
Y CPU-years of processing time
Z TB-year of hard disk space
etc.
I agree with most of your premise, but I think you overstate it a bit. Like the GP's point above, it seems like you're describing bad engineers. In the good web dev agencies I've worked at, they work pretty hard to make informed decisions which stay within budget and timeline. It doesn't always happen, but that's not just due to poor practice.
No industry can, unless it's something that has been done repeatedly before. If you ask a frontend designer specializing in WordPress to make a theme for WordPress and implement it, they can estimate it accurately.
What is thoroughly missing from this debate is the age old project management terms of scope and risks. As soon as you enter unknown land, the risk goes up. And risk means uncertainty and probably expenses. And if the people writing the checks change the scope, the original estimates goes out the window. If you're making a building, it's easier to understand that you cannot just change the scope after the blueprint is made. You have to get an architect to remake it, and everything done and ordered so far may or may not be salvageable. It happens in all industries, but just more so in software development.
they've been waaaaaaaaay longer than CS/SE
they do not change so robustly & fast in short peroids of time.
>We can't even accurately estimate the cost of a complex project given our choice of tools
Let's do not pretend as if that was limited to our industry.
There is still a lot of creative art and the more we stop pretending this isn't the case, the more people will be reasonable. There is still an high degree of technicality in the profession, but that doesn't mean its "engineering" -- you can be technical, not an engineer, and still respectable (heck, look at science in general). It's still a complex skill and makes lots of money for many businesses.
Not really, rapid prototyping - especially due to 3d printing - is a thing now - you can iterate on a model as fast as you can in CS/CE.
Also - a lot of engineering disciplines use simulations - civil engineering, avionics etc - so they too can iterate quickly.
We are absolutely horrible at estimating. I had article somewhere that analyzed plenty of CS/CE projects and found that we are good at estimating median, and if something goes wrong in most complex part - the time to delivery can skyrocket 100x.
Nah, this isn’t really true. Consider all the technology that came out of WW2: digital computers are contemporaneous with jet engines for example.
There are no shortcuts. You have to build a ton of software, and make a lot of mistakes, and most importantly, learn from them. Then there's the other half of the equation and some people just have the creative mind to imagine the right framework, or tool or library or abstraction or architecture for the problem at hand.
Our craft is very bad at providing such a canon. It is even anti-intellectual in parts ("what do you mean you cannot be a master after x weeks? You are a gatekeeper! you suck"). That and we are still not sure what is important and what isn't (i.e. what should be part of the canon). Our medium of study changes all the time. Arts medium of study is reality and reality, for all our advances, doesn't really change.
Maybe related, you were once considered an ignorant brut in this country and others if you haven't read a library of literary classics, now we have kids growing up who barely know Shakespeare, or the rich body of the writing of the abolitionists or slave narratives (how many people you think in this country that even know 1/100th of what Henry Louis Gates Jr knows?) Do you think W.E.B Du Bois and Booker T Washington are common names anymore? I remember when everyone had the preamble of the Constitution and the Gettysburg Address burned into their minds, now we can't be expected to remember questions for a simple AWS Certification Exam, it's pathetic.
I do not believe in the idea of a bimodal distribution of software development (not just programming) aptitude, but I sometimes wonder.
(Regardless, I don't call myself an engineer. Engineers get sued if their stuff doesn't work. I'm a software developer, no matter how good and careful I might be.)
A company/client will often make decisions based on "opinions" too. "We need a website, what should we use to make it?" It's a bit of a blank canvas situation at first where a developer/engineer could come in and say "We should do it X way with Y libraries because <some opinion on why it's the best stack>". Devs could have totally different takes on the same canvas by a matter of opinion.
If I were to ask people "What type of car should I buy?", I would likely get a whole variety of answers from hybrids to SUVs to European and Japanese cars based on who I asked and what their own preferences and tendencies are. I may end up happy or I may end up unhappy depending on which opinion I'm swayed by and how much rigor I put into investigating the different opinions.
Software can be very similar when people providing the "opinions" are caught up in looking to use specific tools over trying to figure out an appropriate solution. When developers/engineers/designers/managers immediate choose tools as a blanket solution over understanding the scope and use-case for context, it's a sign to heavily question the authority of their opinions.
There are a lot of dynamics going on. One is programmer's curiosity but then there is a financial incentive to be on the latest tech. If you see React jobs paying more than Angular jobs, it might be a sign to jump ship which has nothing to do with technical merits. Then there is things like Blockchain that excites investors and techies alike, even when it's a useless solution for something.
People sitting in a room deciding the genuine best thing to do is going to be hard with all these dynamics going on.
We've been building bridges for thousands of years, and still the relatively simple bridge they built down the road from me took 2.5 years to complete instead of the estimated 9-12 month.
I've seen non-science in Healthcare, and I've spoke out against it. I don't really understand why people are stubborn when there is data, and the data is solid.
If I could wish anything in software engineering- follow the data.
reaperducer's rewrite was an improvement, not because he replaced Javascript with PHP, but because he moved all the logic to the server, which is a single environment he controls. He could have used Node instead of PHP and achieved the same thing.
On the other side, when you render on the server, everything is already there. It's easy to write logic when the data you need is an sql query away and all you need to do is spit out some html. What gets a bit complex is keeping working state between pages and this usually requires handling sessions, cookies etc.
If you have a UI, you're going to have to manage UI state somewhere. Even if you have no client-side code, you've marshal that state into strings to send to the client, and then parse it all out again on the server when you get the next request. Remember hidden form fields? Heck, this very site has some pretty interesting history about attempts to solve that problem elegantly. It was a PITA.
Ultimately, none of that has to do with Javascript; it's inherent in a client-server architecture. The one thing that is specific to JS is asynchronicity. If you don't like that, ok, fair enough, but that's a matter of taste. It has advantages as well as disadvantages, and with Promises and async/await, it's actually pretty elegant. It's certainly not the case that asynchronicity makes you write shit code.
> Data being "an sql query away" on the server is similar to being "an AJAX call away" on the client.
This is a bit disingenuous, since sql calls in most back end languages are synchronous. Also an ajax call needs to update the UI to tell the user something is happening. This might mean disabling controls, showing spinners etc. All of this goes away if you only use server-side rendering since form requests are natively handled by the browser.
> It's certainly not the case that asynchronicity makes you write shit code.
But it expands the number of cases you have to deal in the UI, to show all the loading states, error states etc.
I developed my own custom scaffolding for Vue projects[0], which is a lot simpler and customizable. It even allows you to develop Vue projects without a build step/tool, you save and reload. Yes, there are tools that reload the page when you save, but I don't think they're worth the complexity.
It could be further improved to actually compile/build when you actually want to deploy, I hadn't the time or a strong need for it yet. But when you are developing, things need to be simple, I think you need to know everything that is happening in the browser so you can resolve any bug with much more speed and confidence.
Fundamentally, all of it is a matter of opinion. You have a million options and they are all correct. Don't take a million options at once, practice a lot, understand your problem domain. You won't write something perfect every time but you'll write something a whole lot better than if you were using a million frameworks instead of real thought.
In other words, "I can choose the correct tool but still fail from poor approach of problem."
Entire classes of bugs simply don't exist in better languages.
Trying to make web applications with it is another story though. The lack of any state management is a nightmare, hence declarative frameworks/libraries such as React/Angular(JS)/Vue/Svelte etc. etc. came along.
What makes the lack of state management a nightmare in the javascript world, but a complete non-issue when doing .NET or Qt apps?
As someone who started with desktop apps and then moved into web dev, I never really understood why Javascript developers wanted a library to help them manage state.
Meanwhile objects representing various parts of the view, and plenty of models that don't look much like "modern" JS state management, remain the norm elsewhere, and the sky is not falling. I do not know what to make of it. I go along with it when in that sort of company—when in Rome, and all that—but the evangelism can be pretty funny.
Anyway a bunch of the stuff isn't half as special as you'd think reading about it through the usual channels. Redux? A data cache and a so-so local event bus with somewhat awkward semantics and a few really bad terminology choices thrown in. Whee. There's your "state management". Not that it's useless, but you'd swear it's black magic reading all the blog posts about it.
The problem with javascript is that this concept is entirely lost on react developers. Due to the untyped nature of javascript and it's close linkage with html. HTML is made up of dependent components and this essentially puts dependency injection into javascript leading to JAVA like issues that the functional paradigm was suppose to solve.
The reason is rooted in how browsers work.
In native software you can "just" render your data structures directly to the screen. In the browser, barring webgl, you can't render your data structures directly to the screen, instead, you're forced to interact with an imperative browser API to individually manage bespoke UI widgets that store their own state internally and can't understand nor access any of the data structures defined for your business logic. The only option left is to manually update the internal state of the browser UI widgets whenever you update the internal state of your application. Manually keeping the two states in snyc is a real pain, highly bug prone, and a maintenance nightmare. The front-end browser frameworks came along as a way to "auto-sync" the business logic state with the browser ui state.
jQuery just changes the DOM. It gets very difficult to keep the DOM in sync with the JavaScript backed data when there’s multiple moving parts.
Again I don't understand this argument. What's difficult about it? People have been doing it with win forms for well over a decade now.
And yet it's recognized as a problem there, too, and one of the more recent solutions is state management libraries inspired by those used in Web apps, e.g., there are fairly direct copies/adaptations of React and Redux for .NET specifically to address this issue.
le sigh
*And you BETTER be using the minified version for production.
Good or bad depending on context.
Placing the blame on him when you were the one being hostile is hostile.
I think it should be a more widespread idea that people should aim to leave a company with a net-negative `cloc` contribution.
Every system that I've built so far has started out with a simple, obvious set of rules (both new systems and replacements for old systems). Then the customer/PM/whatever notices just one more edge case. And another. And another. And another. The truth is that reality is frustratingly complicated. Especially if the reality you're mapping contains the internal bureaucracies of large companies.
Erase existing software, so we have less amount of shit software.
I'm actually applying that technique at work with great results (really!)
I'm actually more found of when I don't even have to replace it: just delete the damn obsolete shit. Not replace. Delete. Most of the time it for cases where it is just dead. Sometimes it is only quasi-dead (from an incomplete previous modification) and in a few cases deleting it actually fixes some bugs.
The best patches only have - lines :D
The second thing is: companies do not only try to avoid existential threats; they also attempt to have economical approaches. If anything needs to be maintained, there is the potential, if it is crappy, to make maintenance costly.