Scala use is less good than Java use for at least half of all Java projects
goodstuff.im
goodstuff.im
Specifically, Pollak appears to be arguing that a key question in language adoption is the historical or recent win rate for projects. (Ratio of Scala projects that succeeded divided by total Scala projects.)
Pollak wants the average doofus Java developer ("COTM" as he terms them) not to use Scala so that Scala's win rate stays high, even if Scala's absolute number of wins remains low.
But having a multitude of doofus developers leads precisely to things that Scala kind of lacks right now, like quality IDE support, idiomatic libraries for every possible use case, and so on. Without the hoi polloi, why spend a massive number of man years developing IDE support that can efficiently parse incorrect source code in the language as correctly as possible? Why write idiomatic payment processor code?
Similarly, focusing on only the best developers has arguably led languages like Haskell and Lisp to be great languages for learning language constructs and new ways to think, but poor day-to-day languages. (Though I do recognize that Clojure is gaining ground, etc.)
In short, is Pollak right? Is win rate so critical to adoption of a new language that Scala should kick out the COTM programmers? Or is he proposing imposing an exclusiveness that will only slow the development of tools and libraries necessary to open up the language to a larger and larger community?
Sure, their projects may compile, they may even be deployed and used on a daily basis. But are they maintainable? Are they able to function when load increases? Are there hidden race conditions that go unnoticed for years? If a user inputs bad data will the application suddenly throw NPEs? Will moving to a different hardware platform with more cores expose a fatal deadlock?
Scala's "weakness" is not that it is harder than Java, it's that it exposes the difficulty and complications of software engineering to developers at an earlier point in the development cycle.
If your standard of success is compiling and deploying a product, than yes, I'll admit Scala makes it 'harder' than java. However, if you define success as writing defect free, performant software, than I'd argue that Scala is actually easier than Java.
All of these are secondary to turning a profit. If the only developer(s) the business can afford commits these mistakes, so be it; but if the developer is adding more value than he is costing, it still makes sense. And no amount of appeal to technical superiority will change this; there are too many businesses that can benefit from someone able to write software, but not enough returns for them all to hire great devs. This is basic economics. You can't fight it outside of a non-existent world of surplus.
May I cite: http://www.scala-lang.org/node/10714
Kids can learn Scala. It isn't "hard", any more so than the syntax of any language is "hard". It's just another syntax to represent the logic you're looking for. The FP aspect of Scala, imho, actually makes it _easier_ to write this kind of logic.
I think Lift is way, way too heavy a framework. It seems like it's trying to be Scala's version of Struts, which I guess is admirable, but Struts is also way, way, way too heavy. Any time a web framework starts making you learn a brand new templating language, that sets off warning bells. I like the concept of snippets in markup, but I could honestly leave the whole render-logic-inside-class-attribute business.
Here's a group using Haskell: http://cdsmith.wordpress.com/2011/09/03/haskell-for-kids-wee... Are they choking and dying because Haskell is just too magically difficult for children to learn? Goodness, no. On the other hand, does this prove that Haskell is never confusing when you're three layers deep in monad transformers trying to use a parsing library and you get a twenty-line error message about a failure to unify types with type signatures bigger than Twitter will let you post full of "a" and "t1" and "t2" and "t3"s?
I mean, let's be honest, that just doesn't happen when you're using Python or Perl. (It can in Java, but still a lot of times it's just verbose, vs. Haskell's actual complexity.)
Kids can learn anything. The question is, can adults learn it or do we have to wait for the kids to grow up?
My criticism of this was that you don't need to program like him to have a great Scala experience. You can still program in a OO manner and get plenty of benefit out of Scala. As you get more familiar with it you will pick up some nice features that it gives you that Java doesn't. Everyone comes at things at a different level and I think its quite pretentious to claim that most people aren't smart enough to use your tools and they need to stick with simpler tools. If you give someone the tools they will learn to use it. The real problem is when you need to be an expert to use the tools at all. thankfully Scala isn't like that you can jump in with a hour or two of instruction and write perfectly good Java++ code. then as you learn how to use the tool you will pick up the nuances of the language and really unlock its power.
I would say this post like the last overlooks some of the broader issue and really does the community a disservice.
I think it's too optimistic to expect most standard developers (normally employed by companies whose main business is not software) to "pick up the nuances of the language" - these developers typically learn by example and copy-paste-modify from other similar functionality to fill out the application. They see their jobs as implementing features, and understanding nuances is not typically the quickest way for them to get their job done.
In Scala(guessing a bit here since I've not done any large projects in Scala) you might have hidden performance costs, method signatures for your own library code that are insufficiently reusable because of lack of covariant et al specifications.
But the rest of the problems for Scala seems to me to be style guide problems. For example, using or not using . and ()'s, how many classes per file.
Maybe I'm out of the loop, but is it really this bad out there in the real world?
I took some high paying contractor gigs through an agency a few years ago because I wanted to figure out what direction to take in the next phase of my career, the cash was great and I'd never worked in the fortune 500 / government contractor kind of space before.
I wasn't that surprised when I got told that I was now their second developer (out of a few dozen) who "actually writes code in his free time sometimes! isn't that crazy!" since I was prepared for big company suit world.
The reality was right in line with the quote, if not worse.
Myself being a optimist, I don't believe that statement, there is probably someone. It has to be. Then again it could just be my wishful thinking.
I also saw the appeal of that side of the industry. These guys all had way more varied hobbies and family lives since the programming stuff is "just my job" and had a good stable career path because any motivation we in the hacker news crowd would spend on learning new cool stuff they spend on jockying for position in the big office social/political world.
The lure of the office drone never got me though, it was too obvious to me that you are just selling the best part of your day for most of the week during the best part of your life only for money, since the actual hours are spent doing something you are marginally competent at and find uninspiring.
I thought we had gotten past LoC as a measure of programmer productivity?
Hacker News gives us a rose-colored view of the software industry-- one in which knowing about functional programming and agile methodologies (regardless of whether one intends to use them) is a prerequisite for being considered literate, and in which 200 line-per-day development on new invention is something people can actually get paid for. That's now how the average developer experiences software.
BTW, this is one of my startup rules: measure what's important. Decide what's important by how it would affect your decision-making.
However, that depends on who you ask. Specifically, try banks (and other financial institutions), Fortune 500 companies (exclude the more tech savvy such as Google, Microsoft as the code that does get written is meaningful), and anywhere that is infested with suits.
And, by the way: I believe that, out of the 20, 15 are XML.
But using LOC as a measure of work quality or suitability for learning another language is really ludicrous. I would really hope that most people in the industry and Java programmers in particular are capable of learning Scala. I like to think that suitability of the language for a task is a quality of a language not the programmer. Yes, all languages in use are Turing complete, but some are better suited for certain tasks. The question we should be asking is what kinds of projects Scala excels at, and what kinds of applications is it really good for?
And yet think you are part of the group mentioned by the article?
Lines of code is an awful metric of productivity but, yes, that figure of 3250 is accurate. Between 1,000 and 10,000 LoC per year is normal for a software engineer in the industry, and the rate is often closer to the lower number.
Note also that code of average architectural quality will require about 1 maintenance day per year per 80 LoC just by virtue (vice?) of existing. (Paul Graham: a full-time maintenance engineer can own 20,000 lines of code.) LoC are a cost: if we rate a developer's time (including benefits, office space, etc.) at $80/hour and charge each LoC for 4 years, we pay $32 per line of code in maintenance. (We spend $2-3 per line to write it.) That is, IMO, a pretty accurate estimate of what we pay for software.
If you write at 500 LoC/day under ideal circumstances but you're writing code that's merely average in architectural integrity, the maintenance problem will catch up with you very quickly.
It's a lot better to write 150 lines per day of great code (200-300 including unit tests and debugging utilities) and maintain that pace than to start out at 500+ but stall out late in the project. Great programmers don't write code at college-project speeds; they write in such a way that they can sustain triple-digits in perpetuity.
A single developer solves this problem by refactoring the code when necessary and incrementally improving the architecture so as to reduce the maintenance cost. It's only by doing this that projects beyond about 15,000 lines of code per person (75% maintenance burden if code is of average quality) are even feasible.
It's not just a problem of incompetent developers. Maintenance imposes a 70 to 90% overhead on most important projects in large companies, just because these projects become difficult or impossible to refactor and fix and their "broken windows" patterns of decline take on a life of their own. Individual developers can refactor aggressively when their maintenance burden breaks 50% (70% being a crisis) but this is not usually feasible for multi-developer enterprise projects that have changed hands several times.
The more progressive large companies now actively discourage any mention of LoC in performance reviews. First, deleting unnecessary code is one of the best things a person can do for a code base. Second, if someone in a big company generates 250 LoC/day reliably over years, it doesn't always mean that person is a great programmer; it could mean that, or it could mean that he's always taking the fun, "green field" projects and passing the legacy maintenance (on which 10 LoC/day is a typical clip) to others.
This is what I call the Dirty Little Secret of Corporate IT shops: The mediocre programmers spin a story and slag off others to get the jobs developing new systems or rewrites in new technology to pad their CV; the programmers with aptitude end up in maintenance roles digging deep through the said code to fix all the bugs afterwards and make that software usable, while the mediocre programmers who wrote it are already in their next jobs doing it again.
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-d...
This is a problem even when the maintenance programmer is not a mercenary. Check out the essay by Peter Naur (the N in BNF) called "Programming as Theory Building". It's one of the best things ever written about software (albeit in disappointingly bloodless language) and explains this masterfully. http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...
The essential point is that source code and documentation, no matter how well-written and extensive, are insufficient for understanding a large system. You have to have the "theory" (the article explains what this means) and for this you have to have someone who knows the theory.
I wrote a system in C# that used immutable data and actually used that to do rollbacks when evaluating a scenario failed. My fears was that since there are no "val" and really bad immutable collections in C#+.Net the maintenance coder would probably remove list.AsReadOnly() and putting in public on setters as part of fixes to bugs.
That didn't happen though, the maintenance guy was really smart and got the ideas without me explaining much more than mentioning immutability.
But this is why languages like Scala are important. Immutability and similar design(patterns) are a result of discipline and sometimes convention in languages like Java and C#. It's bound to fail due to humans being humans.
So you're saying I was maintaining code that was ultimately enough for four full-time maintenance engineers while developing two games as lead, supporting other developers to put together nearly 100 games based on that library (I was also externally-facing support), AND designing and implementing the beginning of the next revision of the library)?
And people also raved about my support, to the point where it felt like I had a fan club. Maybe the bar is just that much higher in video game development; I'm certainly not even the most efficient developer I know.
I did refactor the code from time to time, though there were a lot of games based on this library by the end that I couldn't risk breaking, so I couldn't change the public API. Mostly I just wrote it to be solid the first time around. It was the fourth or fifth time I'd written a similar game library; after a while you get the hang of it, you know? Not to say there were never bugs, but they were (almost) always easy to find and fix, since the architecture was solid.
My only point really is that rules of thumb like the one about 20k lines of code per maintenance engineer can be off by an order of magnitude or more, if you've got the right code and the right developer. I'd estimate that no more than about 40% of my time at the end was taken up by maintenance, and it was probably more like 25%. And a lot of my "maintenance" time was spent fixing bugs in other developers' games that they just couldn't find, and that weren't in my code at all.
This is why when I interview people who say they've been programming since they were 10, I don't really care - because the experience of working on your own code is radically different from working on an existing codebase.
It didn't typically take more than a day or two to find such bugs, even if the original developer (working on their own code) had tried to find it for weeks and had given up. I could always come in with fresh eyes, and I know most of the ways that people write games, good and bad. In one case I had their "intractable" bug fixed in less than a half hour, having never seen their code before.
So I think I'd probably still pass that interview question, though I'm not looking for a job right now. ;)
(also it means that they are pretty fanatical programmers thus that they're self-motivated)
Yes. Absolutely. That number applies to average code and there are obviously ways to push it out if a person cares (which you did, since it was your project and 4 years of your life were on the line). For example, unit tests add LoC but improve code health.
Game code is...notoriously hard to unit test. We had some functional (?) tests that would do complete screen renders and check the resulting bitmap against a "correct" result, though that was mostly for verifying the code worked correctly on all target platforms, and some unit tests for classes like the string class that were relatively easy to create useful tests for, but far from complete coverage. Most HN developers would probably consider the coverage pathetic, I'd guess, and I rarely even ran the tests (they were mostly for QA).
I've actually never seen anyone create unit tests for actual game code. I've heard about it being done, but it's a rare exception rather than the rule. The religion of "Test First" hasn't really caught on in game development.
Most of the "business logic" (so to speak) changes so fast as you're developing a game it that creating unit tests to verify it doesn't make sense. Also, a lot of the behaviors are "fuzzy" or visual, and unit tests can't tell you if the results feel right -- for that you just have to play the game. Some of the more structural parts of a game's code could COULD stand to be tested, but in general it's not considered to be valuable enough to offset the cost.
I don't know how to count unit tests in such a metric though... because I took a code base from 2% code coverage to 70% and that meant writing something like 200 LOC in unit tests alone per day, even though the actual code being refactored under these tests diminished in size.
Especially in the case of maintenance of existing legacy system, where most of the activity resolves around understanding existing code, writing is just a small part of it. If maintaining the system needs to be financially feasible, hiring expensive Scala developer may not be the best idea to maintain the system. Unless of course, using Scala outweights the benefits of using Java even with the potential financial cost.
This is true in my personal observation. I mostly wrote two pieces of software rougly 50,000 lines each, and it's my limit of manageability. Will need splitting up and modularizing if it needs to get larger.
So the number is really more reflective of how much an organization lumps under "software engineering" and how much general overhead its processes incur than it is of the productivity of individual programmers.
1) Documentation for the standard Scala library is absolute crap. The JDK is much better at not only documenting methods in a reasonable way but also giving you common usage patterns for classes/packages. I really don't like having to piece together the method signatures of a class to figure out the intended usage pattern.
2) This might be a problem in IntelliJ but I can't figure out a good way of getting the debugger to work well. Sometimes it'll catch breakpoints other times it won't.
3)Lack of maturity in libraries. A perfect example is Scala's built in XML support. It works for 90% of what you want to do but finds good ways to waste your time, especially if you're dealing with namespaces.
4) Obscure syntax and abuse of the single argument method call syntax. What does :+ mean to anyone? What's the difference between +++ and ++ or :: and :::. Scala is a dyslexic's nightmare and I don't agree with such heavy use of symbolic shorthand.
Again I love Scala and have used it write some heavily parallel code with minimal effort thanks to the functional nature of Scala but I feel like the learning curve is a lot harder than it should be.
fogus 20 minutes ago | link [dead]
ad hominem rhetoric
Yikes! I seem to have struck a chord with Mr. Pollack. The problem is that I'm not sure that the gap between what he said and what I said is very large. I certainly wasn't attacking him in my post, although I did link to his article. Additionally, I would never question his Scala credentials, but I tend to question those who say that Scala is complicated based on type signatures and syntax oddities. However, I will say in that with his latest post I think the gap between our opinions has widened. His comes off as cynical, but I have to think that I'm misreading the post. I do however find it odd that "ad hominem" would be used by someone who bases a dev's programming potential on the number of hours they choose to work during the day and their familiarity with Zed Shaw.DPP's a freelance consultant, and I suspect one with no shortage of business. He'd rather stick to projects where the technology he is advocating won't get blamed if the project fails. That's not really surprising.
I didn't follow the whole discussion (especially since I was on holidays), but the comments I saw were not obviously worthy of being [dead]; hence my repost.
fogus 6 hours ago | link [dead]
In the case of my blog post I would say you are 100% correct. Without a doubt I was unable to get my point across since Mr. Pollack, whose work on Lift I respect, thinks that I attacked him personally instead of his point. As for my more general tone-deafness and inability to communicate using the English language I can only say that if I keep at it, then maybe I'll one day be successful.
Really? The use of the word "could not" here seems kind of presumptuous.
It's possible that the "barrier to entry" for PHP is simply lower because there are more places you can easily run a "hello world" script, but once someone is setup with some kind of repeatable code -> deploy process "writing" the difference between "writing" one language and another is relatively benign.
He seems to be making out that Java is some kind of hard language to learn compared to PHP.
I learned Java before PHP (and after C/Assembly at uni) but I much prefer writing code in PHP compared with Java - not because Java is "hard" but because it's annoyingly verbose.
Perhaps what he's referring to is not so much people that "code PHP" but people that "get something running" based on existing software or copied and pasted module snippets (like installing Drupal or Wordpress modules).
That would seem to me to be more in the realm of running Macros in Excel or Word than actually being a "coder".
I seriously doubt that there's anyone who's capable of writing a program from scratch in any language that couldn't learn to do it in any other language.
First David writes a post thats not befitting of a person that promotes a language. I am not doubting his competency with Scala - his display of his contributions prove that. But I have found that there is somewhat of an inverse relation between a person being productive in and promoting a language to how much he disses other languages as well as extolling the virtues of his own language. I am pretty sure you will not hear such stuff from people like Martin Odersky, Rich Hickey and Matz. Guido I am not so sure, ;) (j/k, I am a Python enthusiast)
Now that that was said and done, would it have been a big deal to close the chapter and move on?
The only way such a post would have struck somewhat of a cord is if he gave specific code examples for each case as to when scala is hard, or when scala is better. Theres not much of that in the post.
"What has changed is that I've realized that there's a vast quantity of development shops where the developers show up, have a few meetings, write a few lines of code and go home. I've had experience with three instances of that kind of company over the last year"
This is one instance of how off-the-cuff the post can sometimes get. I am sure there are shops like this, but can you draw a conclusion that the above situation equates to what he says in his previous post: "Your developers come in at 9:15 and leave before 6 and don't check work email at night: Hard"
I thought his was the humor section of the post, but I am not so sure now.
I knew of David from a while back when Lift was starting to get popular among people looking for a different web framework. There's no doubt Lift leverages a lot of Scala's strengths. But maybe sometimes (times like this) David should let Lift do the talking.
As I said, simply unfortunate.
I'm having a hard time thinking of a use for the Product interface to Option, but I suppose it might as well be allowed since it is implemented anyway.
Can anyone familiar with this design decision shed light on this?
A good tool should be a force multiplier, not just another damn hurdle.
This pointless way of thinking is very much like finding a more appropriate keyword for a public field, while trying to improve the methods by which VM handles big integers.
Scala is liberal.
It allows you to express yourself more elegantly. But freedom comes at a cost (this is natural!). The price is learning it, remembering it and a constant state of awareness for all the patterns it has to offer.
The speed of a compiler (think of PyPy), is a thing that can be improved at least to some extent. But very few people are willing to work on the inner structure of a language, that you doom to be inappropriate for 50% of the tasks out there.
Look at the set of various languages as a biosphere of evolving organisms. They tend to improve. If there was no demand for Scala or PHP, they wouldn't have occured in the first place.
I think the original article and the follow-up is a poorly disguised attempt at elitism, or maybe just an excellent way to challenge people.
I will say it in simplest terms possible, what really matters is how Scala handles tail-recursion, not whether it allows it. And the first point is out of reach for consideration, if you are a simple developer that may encounter 'learning difficulties'.
The problem with Scala's adoption in my opinion is that everyone, Pollack included, writes about Scala being a "better Java." The Scala books out there, his book included, take that approach. assuming that its audience is a bunch of Java programmers that want a better syntax. "Instead of using this syntax to call this existing Java library, you do this instead:"
So, of course he's going to get frustrated when the Java folks don't "get" it. The article links to another article about the "perils of JavaSchools" which don't teach "real programming."
Scala, like Clojure, strives to be "real programming" on the JVM. It's exciting, and in my opinion, a very fun language and can totally stand on its own.
So, given that Pollack is finding that the benefits are not always there for Java programmers, maybe we'll see a shift in the way Scala is taught or presented. I know at least I would love to see a "so you know Perl, Python, or Ruby? Learn Scala!" approach.
For people without debts to the JVM I think it would make as much sense to just jump to Haskell.
Function1 [-T1, +R]
Then you should be fine[1] Explanation here: http://stackoverflow.com/questions/5277526/real-world-exampl...
The thing about type signatures is that if they're set up properly, you rarely have to think about them. Only when your assumption and the reality of a type system disagree do you have to look up why that's happening. And then you can look up covariance and contravariance is necessary.
But then, that's just the learning approach that works for me. Call it `Just In Time learning or `Agile learning'. `Learn learning', even...
Programming in Scala and programming in Java are equally difficult. That's because the hard part is programming. The easy part is the syntax. If you are mentally capable of "getting" OOP, you are mentally capable of "getting" FP.
If a team decides that Scala is the right thing for their project, then it should be no problem transitioning the good programmers over. After all, it's just programming!
When resources are cheap, things get less risky in a way since people are much more easily replaceable.
Now, I do not condone or even like to work at those kind of places, but that is just the reality in most places.
I have high hopes for Jetbrains' Kotlin project. Even though I don't do any JVM development at the moment if Kotlin turns out to be a good Android language I might pick it up just for the pleasure of programming in such a nice environment.
Those people would do well to emulate DPP's example here and recognize that the long-term success of FP depends on selling to the right audience at the right time and not promoting it as a panacea.