The Python Paradox Is Now the Scala Paradox (2009)
martin.kleppmann.com
martin.kleppmann.com
To the contrary, programmers with the ability to deliver and iterate working code quickly is what is needed -- and especially without overengineering, or inadvertently creating huge future technical debt due to inexperience.
And if there are only 2 or 3 or 5 programmers on my team, and one of them gets sick or leaves, it's far more important to have a big pool of potential replacements who could can get up-to-speed in a couple weeks, not months -- which means a popular language and a straightforward codebase.
The best programmer I ever worked with had an incredible ability to produce straightforward, simple code, that could have been written by "any fool" ... if "any fool" had gotten incredibly lucky. His code looked so obvious in retrospect, like anybody could do it. He made the difficult look easy. I call that very smart, and you desperately want brilliant people like that.
The programmers I've worked with that were traditionally "brilliant" in the sense of quickly coming up with "clever" but hard to maintain solutions only count in the middle of the pack, and I dreaded working with their code. Their code was more likely to not last as long and be thrown away in a refactor.
So I learned to aspire to be more like the first than the latter. And in doing so I saw just how damned hard it is to create simple looking "any fool could do it" solutions to hard or even medium problems. So many times I thought something was clear, but it was not to others. So many times I only finally thought of the architecture that would make it possible to write it simply and legibly after many iterations through other designs that generated much more complex "clever" code. It made me gain incredible respect for the amount of smarts and experience it takes to generate simple, easy, boring code that anyone can easily read and pick up.
E.g. - Our current big project at work uses a lot of Javascript. My coworkers are primarily die-hard Java developers (I know Java, but I don't "live" there...). As Douglas Crockford said in a presentation "[those people] will go to their graves, never knowing how miserable they were".
Rather than writing a bunch of classes with one meaningful method ("Executioner", anyone?), I would rather use FP / higher order functions techniques (e.g. - partial function application, such as in Ramda.js). Also, the idea of anything approaching metaprogramming is very scary to them, rather than pasting in many copies of the same stuff with minor variations. Perhaps it's ("custom") abstraction in general that is scary, since so little is documented in most industry standard practice code.
At what point do you ask your coworkers to realize that the eighties are over, and [static] OOP is not the one true hammer of sole best practice any more???
That requires smarter programmers.
It's just not the kind of smart that correlates with knowing technical trivia or being able to write out a binary tree algo on a whiteboard.
There's a cultural prejudice against overengineering in the python community whereas in the Java community (possibly Scala too) there is something of a cultural prejudice in favor of overengineering.
If you're doing a bunch of "clever" things in your code, you or your co-workers are going to hate it when you have to come back and figure it all out again later.
Smart developers can make the difficult look simple.
https://news.ycombinator.com/item?id=12473688
A relevant section from PG's How to make Wealth [1] essay:
"Use difficulty as a guide not just in selecting the overall aim of your company, but also at decision points along the way. At Viaweb one of our rules of thumb was run upstairs. Suppose you are a little, nimble guy being chased by a big, fat, bully. You open a door and find yourself in a staircase. Do you go up or down? I say up. The bully can probably run downstairs as fast as you can. Going upstairs his bulk will be more of a disadvantage. Running upstairs is hard for you but even harder for him.
What this meant in practice was that we deliberately sought hard problems. If there were two features we could add to our software, both equally valuable in proportion to their difficulty, we'd always take the harder one. Not just because it was more valuable, but because it was harder. We delighted in forcing bigger, slower competitors to follow us over difficult ground. Like guerillas, startups prefer the difficult terrain of the mountains, where the troops of the central government can't follow. I can remember times when we were just exhausted after wrestling all day with some horrible technical problem. And I'd be delighted, because something that was hard for us would be impossible for our competitors."
In my opinion, average-level programmers on critical projects with deadlines will quickly cripple your team's ability to deliver due to technical debt, and throwing 3x more of those easy-to-hire/fire men at it won't solve the issue.
I've chosen such an "esoteric" technology for my startup, because we had ambitious technical objectives, and it proved extremely beneficial - we would never have been able to deliver a system so robust so quickly with mainstream technologies (which I know because I've used those, so I'm in a position to make an informed comparison).
There are legitimate uses of both established and cutting-edge technologies, but from what I've seen the people who fiercely advocate mainstream technologies are more often reluctant to leave their comfort zone than making pragmatic, informed choices.
First, I have yet to see a wave of mainstream technology that lasted more than 2 seasons. Bonus points if it truly solved a problem that businesses have. (Note that this is distinct from solving programmer's problems.)
Looking at my children in the corner playing pbskids, I am struck by how much more productive designers were with Flash and related technologies than they have yet to become with any of the current technologies.
Second, I have yet to be involved with a major "mainstream" technology choice that did not crash and burn terribly. These choices have often ended in public disaster for companies. It has truly become amusing.
Now, does this mean you can't make it work? Of course not. But, with the numbers as they are, realize that even at 1% or less... you actually aren't that unique in the world. But your experience still doesn't scale.
(And I realize this last criticism can relate to me, as well. This is why I won't personally stone wall any choices. I do ask pointed questions about when we will get to focusing on customer's value. So long as we are answering that, I just don't care on the technology anymore.)
Is technology codename for web framework now? How about C, C++, Java, the JVM, Unix, OpenGL, Directx, Windows, .NET, Python? Aren't they also considered technologies?
Oddly, the one that has had the most success is objective c and swift. And that is successful because it is the only game in that town. Despite people trying to make other technologies work.
Compared with how many places failed by picking the new hotness. This definitely includes Java. It took a while before server side became a hit.
So, for example, the mundane way to square a number in Python would be:
def square(n):
return n**2
from itertools import count, ifilter, islice
s = lambda n: sum(islice(ifilter(lambda n: n % 2, count()), n))
It's like Perl golf take to an extreme.
(admittedly, some of my co-workers and I have been known to golf our interview questions into ridiculous one-liners, but we do it for amusement value, not because we'd ever write production code that way)
I joined a .NET shop two and a half years ago. At the time they were targeting the 2.0 runtime, so one of the first things I did was to include LinqBridge for all the Where, Select, etc IEnumerable extension methods. I'm used to functional programming so that kind of stuff is what I use all the time. Not everyone was familiar with it and got clearly confused, but they picked up quickly, saw the utility, and today everyone is onboard.
So, I think there are two sides to it. Of course overengineering without considering understandability and maintainability should be avoided. But you shouldn't shun or even forbid language features because you don't understand them. I know of places where async/await are banned because the programmers cannot be bothered to learn how to use it. That's not what I think the programming profession should be about. C# regularly gets new language features, and I make sure to pick them up and use the ones I find useful. If someone thinks we shouldn't use some feature, I'm happy to have that discussion, and I will abide to what my team agrees to, but the reason better not be "because I don't understand this feature"!
In my professional experience, people that are always learning new languages or are constantly using new libraries and frameworks are never the ones who accomplish the most. They simply are the ones who are always seeking novelty; they never become great at a language, because they are always wanting to try something new. They start lots of projects, but don't finish them.
I don't think those are the best qualities to look for in employees.
If they are doing it to experiment with a new library/framework then finishing may not be the point; it could be to see if the tool is worth the hype or if it adds unnecessary complexity.
This exploratory tinkering has been good for me personally because it has developed my sense for parsing out the bullshit used to market new tools.
There's a huge difference between chasing fashion and seeking out what is new that is actually better.
I agree that those people are usually worse. The ones that pick up on new stuff that isn't necessarily fashionable, assess it intelligently and balance the trade offs of the good vs the bad - those are the better programmers.
IME scala programmers are better - mostly because they tend to be programmers who got frustrated with java's shortcomings and realized scala solved many of those shortcomings, not because it's fashionable to learn scala.
IMO people who chased mongo, on the other hand, demonstrated a responsiveness to large marketing budgets rather than critical thinking skills.
You don't have to try out every new technology either. There are some that you just have to read a bit about to know enough to dismiss them.
As someone with extensive experience in both Scala and Mongo, I agree. Mongo is nice for experiments that might go nowhere, but when you give it a real production workload, it falls apart fast.
It's pretty easy to find people who use "boring tech" and still fall prey to the hazards of overengineering, slow delivery, etc.
If I had to choose between the guy who's done 10 years C# and the guy whose done 2 years C# and 2 years JavaScript, I'd pick the latter all the time.
As a friend of mine put it, "for most people, a 10-years experience is 10 times a 1-year experience".
If the former, by all means, pick some esoteric language. If its the latter, more mundane issues like existing codebase, skillset and, I don't know, perhaps using the right tool for the job? C# can do things Python cannot and vice-versa.
Learning Scala to show off and fit in Graham's paradox is silly and doesn't prove that you are a smarter programmer. A smart programmer is someone who comes up with smart solutions; you can write crap code in any language.
Existing code bases, etc. don't matter so much when you're sharing the same ecosystem, and the good programmers from that ecosystem will certainly choose the best language that they can within it.
But I doubt a moderately complex architecture can live within just one ecosystem. We use PHP, C, Lua (to script the C), Go and Python (being phased out in favour of Go) because different workloads/problems have different requirements.
If you think the difference between F# and C# is just syntax then you have little understanding of either language.
>Whats Python's ecosystem anyway?
Everything on pypi (django/flask/pandas/etc.). It doesn't have another language targeting the VM like C#/F#/Scala/Java if that's what you mean, and there won't be a paradox among programmers of that ecosystem like there is for .NET / Java.
>But I doubt a moderately complex architecture can live within just one ecosystem. We use PHP, C, Lua (to script the C), Go and Python (being phased out in favour of Go)
That mix of technologies suggests that you have a legacy problem (e.g. a CMS based on wordpress you can't get rid of) and technology choices that are fashion conscious (migrating python to go).
I suggest reading Mozilla's story ("from python to go and back again"): https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5...
Little understanding of F# maybe. C# - not after a decade using it! Both languages target the .Net VM - what you can do with one you can achieve, perhaps more clumsily, in the other. My point still stands: the difference between the two ultimately comes down to syntax. And its not like C# is not getting functional (LINQ)
> That mix of technologies suggests that you have a legacy problem
Wrong assumption. Don't assume that everyone write web apps.
We're in network security and need C for high performance DNS server code. PHP is to serve our admin dashboard (not Wordpress!). PHP is fast, and works, and there are programmers for it.
The transition Python to Go is an example of business considerations taking precedence over "fashion". Python was great in the early days when shipping and iterating fast mattered and we had a small customer base. But we always knew we'd outgrow it - Go is a great language: fast, concurrent, parallel and more disciplined. We need all these now that we have large customers and a growing customer base. Business needs drove the move; I am very pragmatic about these things. C++ programmers "get" Go, the transition is fast.
Ironically, I find your comment makes assumptions that are "fashion conscious" - PHP is not some legacy language whose only reason to live is Wordpress. C/C++ is a great language(s) and perfect for certain tasks, but Go is increasingly encroaching on its turf. In future we may rewrite some of the C code in Go - the lack of control over GC/GC pauses/freeing memory is the main obstacle to using Go for certain performance critical tasks.
Maybe my meaning was not conveyed clearly enough; I was speaking within the context of the thread which was narrower. In practical terms, F# and C# both target the .Net VM.
A more apt comparison would be between, say, PHP and C#, or Prolog and C. All Turing complete, but its not practical to use one instead of the other for particular tasks; its highly impractical to write a device driver in PHP.
The same can be said of other languages with less emotion surrounding them. If C++ is the right tool for the job, don't use C just because of ideology. You are running a business, not a religion. You can only afford to be choosy if there are multiple "right tools for the job".
Not at all.
If a tech - at any stage - meets business goals, then go for it. Having said that, its fair to say that early stage tech has risks (project being abandoned, etc) that do need to be considered.
I think my choice today for a Python Paradox would be Erlang or Ocaml. Ocaml + MirageOS is a system level simplification that makes application correctness/security a much more tractable task than [any other language + linux].
Erlang is the most common sense way to build resilient, distributed applications. But there is a high barrier to entry to learn both Erlang and the OTP framework that makes it useful.
Having said that, I've never worked at a company where choosing Erlang or Ocaml was a possibility. I think there are companies that make bold technical choices, but I haven't had the good fortune of being given the chance to work at such a place.
But I agree that the Bay Area, and San Francisco specifically, is a bit of a bubble. So its very possible that there is little Go traction in other cities.
Is it still truly mainstream, or have C++ users, at least on Microsoft, mostly migrated to managed C# code?
Certainly at the company I work at, there's been an active migration over maybe the last 5 - 6 years to C# from C++, though we still have millions of lines of C++ / MFC / WinForms code in the primary application I support and help maintain.
Career-wise, I'm quite happy to stay with C++, as it's been my experience that I can throw together utilities that run with minimal, run fast and are reliable, just using C++ and the standard libraries.
As far as writing Scala or Python, Why write code that goes slower than it needs to?
Let me put it this way: why write software that goes faster than it needs to but is written slower than it needs to be?
I've been easing into the C++ standard library, lambdas etc., and familiarity makes it easy to write code that uses standard proven, fast, data structures, and mostly allows me to avoid any non-stack data allocation.
Avoiding heap memory allocation, and taking advantage of the appropriate idioms, allows much faster development, even in C++.
The huge bonus when done correctly, is very high-performance, clear and portable code.
If you filter for Scala, you are more likely to pick academic purists trying to show off with their core. If you filter for go, you are more likely to pick pragmatics, who will be aware that every choice is a trade-off (including no generics) and will value maintainable code over clever code.
I already made a choice.
I would consider a more useful filter looking at people that have experience in multiple levels of the stack, whatever your particular niche happens to be. For web dev, having a good handle on two or more of: SQL, a server-side language, and JS/CSS/HTML. Substitute whatever blocks fit in your particular equation.
It is enough to be fluent with a common framework such Play? Must one be proficient in using it in all three - functional, object and procedural programming contexts? The breadth of the language really is an overwhelming barrier to entry to someone coming from say Python.
I am interested in hearing what people think.
by number of jobs, scala is way ahead, also python, javascript, ruby, java.
Even here, when jobs are listed, numbers of go jobs are quite low
Stat usage wise it makes Haskell look mainstream... but serious real world code is written in OCaml (I would argue maybe even more serious than Haskell :P ).
The users of OCaml almost always know how to parse really hard things to parse which is a pretty descent filter.
MirageOS sounds cool. What are people using it for?
It was specifically design to disallow smart-asses to write ninja-code, so no. Go is for monkey programmer factories which managers want to make sure they will only code a certain way because Go doesn't give the developer choices. It's the anti C++,scala,python,clojure... Go tells you to do this that way, and doesn't give you the constructs or features to make your own choice about coding.
I know that doesn't sound fun but it makes maintenance so much easier.
In Scala, you basically have to learn the constructs and their semantics. No way around that. But once you do, almost everything is just various compositions. For the most part, it's very predictable. The execution model is ultimately pretty simple.
Consistency -- I'll give you that. There are a lot of options for how to structure your code. And when you start composing libraries together, you have to be able to understand the paradigms the authors use.
Operator stew, whether Scala or C/++, REALLY makes me appreciate the Lisp style of function calling. "Evaluate inner parentheses first. There is no step 2." Now you're trained in "operator" precedence, which, by the way, I have seen responsible for far more costly errors than runtime types. (I'm putting C's lack of any real type checking at all in a different level of problem - sometimes useful for systems stuff, but foolish for general biz apps)
Today, the story is much improved. SBT has been massively reformed, with insane operators deprecated.
http://erlang.org/doc/apps/dialyzer/dialyzer_chapter.html
http://elixir-lang.org/getting-started/typespecs-and-behavio...
Not the same as TypeScript or Haskell, but may help.
Note that Erlang/Elixir programmers don't care about most runtime errors as we use Supervisors to restart processes that may crash because of them.
That doesn't give me a whole lot of comfort. A program that handles crashes gracefully doesn't fix its own mistake. Supervision is great, but it's even better to not have the crash in the first place.
Note that "practical" functional programming is something I just made up, but in general to me it means a language that provides immutable data, first class functions/higher order functions, and provides ways to minimize side effects. Elixir/Erlang do these. The language is geared towards functional programming, but it's impure.
But compare it to Haskell, a "pure" language: Elixir has no strong types, monads/functors/burritos, you can do side effects whenever you want with no penalty, there aren't many restrictions or things the compiler will yell you at about that aren't obvious mistakes.
I'm not saying either choice is better, but the average developer with little experience in both Elixir and Haskell can probably build a product faster in the former.
This mattered much less in the past as ecosystems were much smaller and people tended to build software much more from the ground up.
That said, I think it applies within ecosystems still. Scala programmers get to choose a better language without abandoning the Java ecosystem. F# programmers get to choose a better language without abandoning the C# ecosystem. The "paradox" is evident there.
Go is definitely not that language (generics? lack of package management for years? c'mon). Rust, maybe (for systems programmers).