There is art to it. I wonder if it can be taught.
I'm of the school that thinks it's something you need to practice. Many software positions especially in large companies are very limited in scope, you basically just maintain and patch existing systems, rarely having to build systems from scratch. Building these systems from the ground up is what is hard but you learn what works and what doesn't the more you do it.
If you want to learn how to build software, join a start up or small company that has a healthy appreciation for what software rewrites can provide. Run away from companies that follow the never rewrite mantra. Usually saying something like you'll just repeat mistakes, its like well no shit this thing was built before I was born, if you rebuilt it with me here I could avoid those mistakes.
I'm not saying you should always rewrite, its usually better to refactor, but some design failures are hard to refactor out, have double binds or are too critical for stability woes. But systems that live forever should have unlimited quality assurance budgets for refactor and repair, very few do in this industry. Managers seem to expect systems to live forever all while constantly patching with very small refactoring budgets. It's completely insane.
Like running a machine in a factory 24/7 and never doing maintenance but expecting it to work perfectly like new forever with no future cost incurred as eventual debt.
Preach it! I don't know where this mentality comes from, but it seems to treat software as something that just magically solves problems and only needs a little upkeep to chug along.
The problem is that problems change, and a different problem is no longer necessarily a solved problem. It takes a lot of design effort to make something that can easily be adapted to changes in the problem space, and it still requires effort to make sure the software doesn't specialize itself over time into a corner.
Thinking about the above further I notice the things I've learned over the years also require the ability to link problem to cause, or in my case it came in reverse, cause linked to problem.
I ran into problems for years, learned how to fix them, and then during code reviews (even in self review), discovered the cause behind these problems.
Sometimes these were causes that would only become problems at a later date because of knock on effects.
I don't know how you teach that. Possibly antipatterns / code smells which usually hint at underlying problems that may be elsewhere / at a different level.
Refactoring by Fowler and TDD by Beck cover a lot of these in good detail.
The examples are kinda dated but the approach is pretty timeless.
Refactoring starts from first principles, almost all examples include the reverse refactoring as often you need to undo optimisations a few steps before you can move forward in a different direction.
I was recommended working effectively with legacy code by the same types of managers that expect forever returns off the same code. They like to point out how that book encourages small patching without refactoring as refactoring is often fruitless in short term money terms, whereas Fowler tends to advocate refactoring as a healthy and necessery part of software development.
I can't say I hated working effectively. I found TDD by Beck more useful, and working effectively with unit tests to also be worthwhile.
All of these books cover a lot of similar ground.
If you're going to read only one or two I suggest TDD > refactoring and avoiding the others.
(Just my opinion based on what I got out of them reading after 10 years on the job, I suspect juniors might have a different take)
However, I have ended up inheriting a pile of crappy code driving business value a bunch of times in my career (data science is much worse for this), so for me, Working Effectively was really really useful as it provided ways to deal with the craziness.
As soon as I finished WELC, I ordered refactoring, and devoured it - but it is less practical for my situations, where there are no tests and nobody sees why they would even be useful.
In terms of good and impactful reads, TDD is probably the most bang for my buck I've ever gotten, in that it's super short and very very good. Kent Beck is hilarious also, which definitely helps.
Of course it can be taught, it has to. The current model of expecting people to just pick it up on their own by going to school or a bootcamp isn't working.
I think a large part of the problem is that in contemporary workplaces it's becoming rare for senior talent to spend serious time developing juniors and doing so as an integral part of their job rather than merely within the scope of some mandated "knowledge transfer" phase.
To many places are beset with pushy "deliverables" treadmills and suffocating project management. It's hard, especially in large organizations, to just "think" and be creative. Juniors need to see that, emulate it, and get meaningful feedback beyond a dumb burn-down chart.
I was an engineer before switching to programming and despite throwing the word Software Engineer in my title, there is no Engineering.
There is developing only.
'best practices' come from Authority, not science(note at bottom). This is a stark contrast to Engineering where everything is proven with math.
(Note)We don't write in logic gates anymore, efficiency is obscure at modern coding levels.
You've just described the field of formal methods. For software with strict requirements of correctness, correctness is sometimes proven mathematically.
It's not that a rigorous approach to software development is impossible, it's more a question of when it's really worth doing. At present, formal methods are very costly to use. I don't think it's always necessary to go all the way to full formal verification for software development to count as rigorous, though.
Related reading: They Write the Right Stuff, from 1996. [0]
> efficiency is obscure at modern coding levels
Depends on the domain. There are plenty of domains where efficiency doesn't much matter, on modern hardware. Game engines are still tuned for efficiency, and will continue to be.
[0] https://www.fastcompany.com/28121/they-write-right-stuff
You can modify and time, but you can't know ahead of time.
In Engineering, everything is defined with specifications and based on specifications you can calculate if something will work. If you need something to work at 100 degrees C but it will never get to 110C, you don't buy a material that works at 110C, you find the cheapest material that works at 101C. (Factor of safety aside)
You can prove in math your design will work
In Programming, unless you are working at switch levels, how could you prove your code is most optimized?
It's not going to be. Abstraction has removed the possibility of doing that.
The closest thing I've seen to Engineering in the Programming world is Industrial Engineering. This is the way Engineers optimize production environments. You can see this in Programming in various time/load aspects. But industrial engineering is an "After the fact", and as much as I like optimization, it has a reputation of not being Engineering.
Hope that makes sense. Science and math vs optimization.
It's possible, but extremely labour-intensive, to mathematically prove the correctness of a program. I mentioned this in my previous comment.
> In Programming, unless you are working at switch levels, how could you prove your code is most optimized?
'Most optimised' is an entirely different problem than 'your design will work'.
Complexity theory lets us do something like this, but at a more abstract level, rather than at the level of real-world performance on a particular machine.
As you say, real-world code is almost guaranteed not to be perfectly optimal in terms of performance. We really never need to aim for this though. Market forces push for high correctness and performance, to varying degrees across different different problem domains. Performance matters in game-engines and for high-frequency trading, and in those applications, the software engineers put in great effort to optimise their systems, using fewer abstractions.
Sometimes the software engineering challenge may be a life-critical hard-real-time system, such as in medicine or aviation. Software engineers are able to deliver such systems. It's not of great concern that these systems aren't perfectly optimal in terms of their performance, provided the programs give the right outputs and always meet their deadlines.
The generation of perfectly optimised code has been researched, branded superoptimisation, [0] but it's little more than an academic curiosity. I doubt it will ever be possible to scale it up to be practical for large programs. (I'm not sure if/how they deal with the way most modern processors are terribly complex and don't have easily predictable performance.)
> Abstraction has removed the possibility of doing that.
It's very often a sound decision to trade off on performance in order to gain on some other dimension, such as development velocity, or maintainability, or indeed correctness, given the project's resource constraints. As tools improve, the Pareto frontier advances, and we get a little closer to being able to 'have it all'.
An example of this might be the use of Rust for web development work, which can apparently greatly improve performance (or greatly reduce the computational resources needed). It can bring the performance of C++, while retaining many of the advantages of safer, higher-level languages like Java and C#.
Whether it does this well enough to really grain traction, we'll have to wait and see, but I think it's a good example.
> The closest thing I've seen to Engineering in the Programming world is Industrial Engineering.
Industrial engineering is an entirely different discipline than software engineering.
For examples of 'proper software engineering', the obvious candidates are avionics (as discussed in the article I linked above), and development methodologies involving formal methods, such as with the Tokeneer project. [1][2]
[0] https://en.wikipedia.org/wiki/Superoptimization
[1] https://blog.adacore.com/tokeneer-fully-verifed-with-spark-2...
[2] [Huge PDF] http://www.adacore.com/uploads/downloads/Tokeneer_Report.pdf
I'm going to highlight this-
Engineering is Science and math vs Programming which is optimization.
Again, mathematical tools can be used to prove the correctness of programs.
With solo programming, everyone has to figure everything by themselves. A deeply inefficient system.