Programmer as wizard, programmer as engineer (2018)
tedinski.com
tedinski.com
However, when it comes to code we treat it similar to writing. We may have a first draft, but the final draft is often nothing more than a cleaned-up draft. I could be wrong. I never wrote professionally.
It would be interesting if we had languages that would be great for prototyping but designed to be unusable in production. However, I'm having a hard time imagining properties that don't already exist in languages like Python and JS. You want weak typing of course, but you'd be ok with poor security. Maybe we'd some nice features that would make the language run slowly since it running in prod would be a non-goal.
But my "notation" here is always miles away from what code would look like, and I think "sketching" in pseudo-code or real code would fail to provide this same springboard. One of the biggest things with writing code is that data structures (especially in the form of objects) quickly lock you down to a certain design and it becomes progressively harder to think of the problem in any other way than your initial view of it. Plus, the hard parts typically require huge amounts of infrastructure to be in place before you can run them for the first time, which works much worse than an abstract brain model in my experience.
>It would be interesting if we had languages that would be great for prototyping but designed to be unusable in production.
The reality here is that people would find a way to make it usable in production and this vision of a good first draft would quickly fall apart :/
For simple code I make type errors into runtime errors and let the compiler figure out type signatures. This emulates dynamic languages with a great linter pretty well.
For complex code I write rigid type signatures first and then play type tetris. This works best with dependently typed languages and is called type driven development. Idris can fill in large parts of the program with this - since type signatures are equivalent to propositions and programs to proofs this is basically proof search.
The issue is that in art or industrial design, someone can't reasonably sell the prototype as a completed work. With software, particularly anything that is web-based, as soon as a sales person sees anything that vaguely resembles something they can sell, they will sell it. Sales means bookings/income, so that prototype becomes the MVP.
Ideally the final draft is something that has been very aggressively refactored, multiple times, with input from a Refactoring Engineer.
(Did I just invent a new job category? I don't think there's currently any equivalent of an Editor in the Software Engineering world; code review is a chaotic approximation.)
Unfortunately there is usually pressure, and maybe also desire, to just make it work, maybe with tests, and move on to the next thing, at least in companies.
In essence I agree with you that this should be an on-staff role, but perhaps the reason we are not seeing this is that the job is usually fairly quick (a couple of weeks/months) and for the business its hard to justify a full-time staff member to perform this service. There is also something to be said for how people in this role are able to gather a wealth of experience by working on a high number of code bases instead of being stuck continuously working on the same few software projects all year.
- the mythical man month
-- Management
You've never seen a "//I don't know why this works, but don't touch it" in code?
Front end JS and TS went through a phase of rapid change 2 to 6 years ago, but I don't think that rapid change is still happening. Certainly there are new libraries, and maintainers still like to play fast and loose with backwards compatibility, but it's not obvious that you have to keep up to date with every single new framework anymore.
An equivalent but probably equally not-the-real-explanation argument would be that we've gone from an era of opportunity into an era of oligopoly as the internet titans have emerged and so the "coolest kids" who everyone cargocults have gone from being fast-growing startups to members of the big-5 elite. Kind of the same as his argument just with detail on who his "people" are and why, but it changes the implications.
I think it's more as he touches on that the two are converging. I think eventually some sort of pluggable typing will win (the "proofs" on your system won't be a single compilation pass but maybe different typing for different parts of the system, and specific proofs run between compile and runtime as the two blur), which will look more like gradual typing.
Most systems change all the time. So long as a company needs executives to make decisions and steer in different directions depending on the economic conditions, companies also need the flexibility to change their code.
Even the Linux Kernel which is now decades old is still being changed all the time. If you use tools which assume that every line of code you write is not going to change, then unless you're programming an aircraft or a medical hardware device you're probably using the wrong tools.
You should not assume that just because some low level module is deeply nested within the code, that it means that it should not be changed or thrown out.
That's why I prefer dynamically typed languages for web systems; they start with the rigth assumption about the ever-evolving nature of the project.
If JavaScript was designed to be statically typed from the beginning, web browsers would not have attained the usefulness or popularity that they have today.
You can partially provide the same benefits in dynamic languages with tests, but at that point you're paying the same cost as with a type system.
Ideally, types should not traverse too many files; there should be a clear class hierarchy and each level should provide more abstraction. Also it's better to encourage passing simple types like strings, numbers or clones of objects instead of active instances... Because if an instance of a class is referenced in many parts of the code, it's difficult to track which part of the code was responsible for changes made to that instance; also it's harder to maintain your train of thought when traversing many files to debug a simple operation related to a specific instance type.
On a related note, I've noticed at multiple companies that when they force developers to use specific IDEs which make it easy to find stuff, the directory structure of the projects tends to suffer (since existing developers don't rely on the directory structure to find things; so they stop caring about it) - This makes it harder for newcomers to make sense of the code and makes the project totally dependent on the IDE.
Also, I'd go further than "pass clones of objects instead of active instances" all the way to "only use mutable objects when performance demands it, and keep the scope of the mutability small".
I agree with your assessment that we need to design for evolution.
I disagree with your assessment that dynamic languages provide that. Dynamic languages certainly do hit a sweet spot for fast iteration on small projects. As soon as your project gets large or long running, the assurances that static types provide get really nice. Witness how much effort has gone into building type systems for dynamic languages: Typescript (MS) is exploding and had vigorous competition from Flow (Facebook). Python has mypy plus a bunch of corporate backing: MonkeyType (Instagram) and PyAnnotate (Dropbox).
I feel much more comfortable aggressively refactoring in static languages.
At the risk of being inflammatory, I would say that you've been looking at poorly designed systems.
A running joke about Google engineers is that all they do is shuffle data from one ProtoBuf to another. It's largely true. The upshot of that is that every single service/system has common, typed definitions of the data structures they use and require. Data never drifts out of sync.
To be sure, a lot of the problems I dealt with there were around data structure migration on large systems — but there was never any uncertainty about the structure of the data itself.
I like JSON APIs, but I think JSON especially contributes to people getting hand-wavy and casual about the structure of data across systems in stupid ways (see NoSQL vs SQL).
There are a lot of overlapping distinctions, some blurry, I think. He chooses some for wizardry vs engineering but I don't know whether they're best. For example he puts "magic"=implicit, but I would normally say highly implicit code is "engineering" because it implies that someone has understood and explored the problem enough to write a very tuned framework or library. I think reasons for putting very implicit code in with wizardry could be that it might be that the person using it doesn't understand it, and that it's somewhat in opposition to strong typesystems
wizardry/engineering high uptime/downtime possible efficiency important/efficiency not important large codebase/small codebase not implicit/implicit (I disagree most with this one) problem is understood/code is exploratory (I think this is the most important distinction) high specialism/low specialism of coders changes slowly/changes quickly typed/untyped
I would say websites are engineering because the problem space is well understood, they need high uptime and they tend to be written by specialists (React guy, django guy etc).
The way I think of it is "wizardry" has good initial velocity (as it relates to change), but velocity decreases with time, whereas for "engineering" velocity is terrible at first (initial investment), improves a bit over time, but most importantly it settles to a constant level.
Engineering is O(log(n)), wizardry is O(N*N).
The inverse can be said: a statically typed language makes it easier to replace parts later, do automatic refactorings, etc., and know you got everything right.
Dynamic just let you have less ceremony (no type definitions) but forces you to keep this all in your head (you still don't want to pass the wrong type of thing to the wrong receiver) and doesn't give you any assurances.
With type inference and/or autocomplete, statically typed languages are faster to write and easier to get right than dynamic.
Albeit, rather than "wizards like implicit/magic, engineers prefer explicit/boilerplate/maintainability", the difference Yegge suggests was management of risk.
(Also a reminder that we're probably going to lose some interesting stuff when Google+ goes kaboom...)
But when I miraculously was called upon to maintain an enterprise code base in Common Lisp, it was an absolute joy. Because whenever they encountered a roadblock in maintenance, the Lisp wizards who had come before me just wizarded up a solution. One of the things that stuck out was that it had its own custom test framework, that was head and shoulders above XUnit, Mocha, or any other commonly-used test framework. Adding a new test was virtually a one-liner; the test would generate test data , send it to the server, and check the server's response against an XML template provided by the test case.
Mostly because
- Clojure is very very terse
- Java has the battle-tested libs
- they run on the same (J)VM - so no FFI required in your code
I think that is a reason why there's a lot of python/c++ in hedge fund land. I've written some clojure but don't know the c interop story for it.
You just opened the box of pandora for a million reasons most of which are unbeknownst to the both of us :) But for the sake of argument I will continue under this assumption
> language to be about as fast as possible
If you're interested in Mathy stuff like Machine Learning, FFT, ... then maybe. But even for those you usually have JNI bindings, so it's easy to use most of those mathy C libs if necessary.
But I guess that 95% of all software isn't about speed but about something else (correctness, maintainability, safety against threats, portability, ...) because costs today are usually dictated by manpower costs or those arising from safety/security incidents and much less often by hardware costs compared to say a decade ago.
The Python libraries never seem to care. Just give them a list of a list of number-ish values and off they go.
Java might have some great ML/Math libraries, but the fact that Python dominates data science suggests that my experience is a real-world pain-point.
For me, it's "simple": I never ever ever ever write one to throw away (and I do mean that in a literal, absolutist sense, which is rare for me). This does mean that I often can't use wizard tools (so it can be less fun to build). I never use dynamic languages, and build on the JVM (Scala or Java), because I know it will scale, and there are battle-tested libraries that do nearly everything under the sun floating out there. (If your org has a different "blessed" platform for production services, then use that.) It isn't quite as quick to build the MVP as if I was just hacking something together. But I can do it fast enough, and still end up with a maintainable, evolve-able code base.
It's not a perfect process. My first version usually has only a few tests that verify behaviors that I had trouble modeling clearly in code and didn't feel confident about. Sometimes I miss error cases here and there that someone else has to find and deal with later. Also note that, because I use strongly-typed languages, I can push a decent amount of correctness verification onto the type system, so the compiler catches a ton of errors that I'd need a giant test suite to catch using many dynamic languages. The tests that I do write focus on logical correctness, not code correctness.
But at the end of the day, I deliver products on time that I feel much more comfortable being robust in a production environment than build-one-to-throw-away prototypes. Stuff that I'm fine holding a pager for if I need to. I have several "prototypes" that are still running in production several years after my first release, maintained by other people after I've moved on. And by and large they still contain a lot of the original code, and the design remains close to (and/or continues to be heavily influenced by) the original design.
On the flip side, I've had to deal with code that's been thrown together with the expectation that it could be thrown away later (of course it never can be), and it's incredibly difficult to bring it up to a robustness level that would be deemed acceptable for a generally-available product. These code bases constantly set off pagers for dubious reasons and write unactionable crap to logging systems... and it doesn't have to be that way!
I encourage you to consider trying out more throw-away work before writing the real thing. You find yourself with not only a more lucid vision of what goes in to the "real thing" but a little more muscle memory in getting started down the right path.
(Even something as "simple" as email.)
The best you can do is paint the fresco on paper, digitize, design the building outline and iterate 3D building designs. And you have not considered materials and structural design at this point and do not even have a scale physical model.
Prototypes work for small, enclosed apps with limited functionality. Not even video games most of the time and these are relatively small and specific. Word processor? Good luck. DAW? Oh my. CAD/CAM tool? You've got to be kidding me. IDE? Nope. A compiler? You'll rewrite it a few times. Desktop environment? A good recipe to lose users. Even CRUD...
You cannot iterate a painting into a building with a fresco.
The thing is that currently an MVP is definitely huge, as much as startups would like everyone else to believe otherwise.
Everything Rich Hickey has been giving talks on and clojure itself seem to be the best _solution_ I've seen.
> People have certainly managed to create test suites that make it harder to maintain the code.
Sure, but pretending that this is the norm is just not true. And arguing against an extreme case can be done against anything .
> I get the impression Google has been able to migrate a lot of C++ and Python to Go using this approach.
Not only is this heavily suspect (I'd love to read a citation for this) but if there's a language that represents the "engineering" side (?) pretty clearly, it is C++.
> Gradual type systems has started to garner a lot more interest.
No they haven't. Those have existed for over five decades . Another problem with these articles is how they seem oddly ignorant of the history of computer science. This is specially odd coming from a PhD (assuming that I looked up the right person in google).
And, last but not least, some of the best engineered pieces of code are precisely shells, dynamic languages and frameworks.
It's things like these what makes these articles seem like they were written by java developers annoyed because java is no longer the trendiest toy.
There is nothing in "started to garner more interest" that implies something is new. It's even the other way around.
Note: This article was not written with Machine Learning in mind, and I will have to re-read the article to better articulate my thoughts on “Machine Learning Wizardry” and juxtapose my own ideas with those of the article author.
Kudos to author: The article’s main metaphor is excellent because it got my creative juices flowing (i.e., brain working at 110% for a few brief moments).
I've read just here on HN that microservices architecture is none other than an implementation of the original concept of OOP.
Suppose the thing you are dealing with is “Customers”, then with OOP each Customer is a separate object, while with MSOP you must communicate to the service which Customer you are talking about. The thing you talk to and the thing you are talking about have different identities.
But with the speed at which the sheer amount of new software 'stuff' comes out each year, there's simply hasn't been enough time to develop rigorous engineering specs or best practices for all of those tools.
Perhaps we're starting to see an initial version of a universally accepted model, at least for the frontend, with the mentioned Typescript + Flux (I'll just say Redux).
Many assume that Redux is only a state container, and it is at face value, but more semantically it is (when implemented correctly) a very logical boilerplate where it puts everything related to state in its proper place, so any developer can look at the code base and immediately get a general idea of where state is set, what events exist and how the state is used throughout an application.
I'd like to see things like these very strict patterns emerge for other tools, like Node. For example, I could imagine a snippet of THE universally accepted express boilerplate for a login, given a specific backend... (can an email + password login really get _that_ customized?)
...argh, on second thought I suppose it can, but even then such a boilerplate could have sections with a freebie spot for customizations
Eh, it's late and I wonder if such universal pattern ideas are a pipedream... I suppose only time will tell...
What is needed is an artifact like spellbook, that the wizard when faced with a situation could describe it to the spellbook and get back the correct spell or combination of spells to solve the situation. Attempts have been made to create such an artifact, but unfortunately the resulting spellbooks still take a long time to find the correct spell and when reading the spells you often find that there are missing ingredients or a complicated set of gestures that must be performed to make the spell work, and you have to read all about these gestures in turn to figure out which ones you really need.