State of the Lambda | Hacker News Reader

State of the Lambda

cr.openjdk.java.net

63 pointsjondot43 comments

> An alternative (or complementary) approach to function types, suggested by some early proposals, would have been to introduce a new, structural function type. A type like "function from a String and an Object to an int" might be expressed as (String,Object)->int. This idea was considered and rejected, at least for now, due to several disadvantages:

Too bad they didn't, C# originally had delegates that you had to declare, but with the addition of generics and lambdas, that system got prettier and easier. The generic delegate types Func and Action are just that, so I can have a method that looks like this:

  private Foo MyMethod(string str, Func<Bar, string, int> func) {
    //...
  }

...which means that MyMethod is a method that takes two arguments, the first is a string, and the second is a method that takes two arguments, a string and an int, and returns a Bar object. But when I call it, I can use a lambda expression, like this:

  var result = MyMethod("hey hey", (s, i) => new Bar(s, i));

...which is far from unwieldy. But I can see how checked exceptions would make it horrible, and I'm glad C# doesn't have those.

Using lambdas in Java 8 looks pretty much exactly like your example.

The line you quote from the article is about a proposed way of declaring lambda types, where instead of

    private Foo MyMethod(string str, Func<Bar, string, int> func) {
      //...
    }

you could write something like

    private Foo MyMethod(string str, (string,int)->Bar func) {
      //...
    }

> It is our intent to prohibit capture of mutable local variables. The reason is that idioms like this:

  int sum = 0;
  list.forEach(e -> { sum += e.size(); });

> are fundamentally serial

An unfortunate decision. Yes, updating such a variable from multiple threads can cause race conditions. But there are many ways to run into race conditions when writing multithreaded code, and this doesn't really add a new one.

Furthermore, most code is mostly serial, and the ability to modify captured variables is a very useful one in serial code. (I've written code like that in Lisp for decades.) And finally, the restriction is, of course, easily worked around -- just store the value you want to update in a 1-element array.

So the restriction doesn't really protect anyone from anything -- it just forces them to use an ugly workaround in order to write their algorithms in a natural way.

Ironically, "sum += e.size()" is conceptually not serial at all; the order that each iteration runs is not relevant while you could restructure the computation into a tree of partial aggregations.

The work around in Java, using a Cell<T>, is a hack that gives you the updatable mutable variable you need without too much ugly overhead. so something like...

  cell = new Cell<T>(); ... { ... cell.Value = ... } })

The motivation is that the question of whether list.forEach (or maybe better, iterable.forEach) is serial or parallel is to be left up to the implementation of the collection. Everything related to multi-threaded code since about Java 5 has been about hiding the details of concurrency and allowing programmers to think about code in a serial fashion, but still get the gains from multi threaded code.

It's not necessarily about parallelism either -- you could just as easily pass in a function as some sort of callback that gets called far in the future, for example,

String error = null; return createClientWithErrorHandler(e -> { error = e; });

What does that even mean? Does the execution context need to be kept hanging around?

Java already provides good atomic references if you need to have a sum variable or anything similar.

> What does that even mean? Does the execution context need to be kept hanging around?

Once the block containing the declaration of 'error' has been exited, the only way anything could refer to 'error' would be if there were at least one additional lambda expression in the same block that was closed over the same binding of 'error'. Two such lambda expressions could thus communicate with each other through the shared binding.

I'm surprised more people aren't just using Groovy. The performance of Groovy 2.0 on JVM 7 is greatly improved to the point where its a non-issue in 95% of situations (and you can just drop to java for those portions with no penalties).

I tend to agree with you, but Groovy 2 is still pretty new. In my experience, most of the hardcode Java shops and devs that rejected Groovy (primarily?) because of speed are generally slow-adopters, and wouldn't be using Groovy 2 just because it was released a few months ago.

As to speed, I've not benchmarked it on JDK7, so I'm not sure what effect the InvokeDynamic stuff has yet, but @CompileStatic annotation helps a lot.

My fibonacci stuff at https://github.com/mgkimsal/newgroovy2 (did a presentation on groovy2 last week) showed:

fibonacci run of 30:

* pure java (not in the github) - 12ms

* groovy2 compilestatic - 18ms

* groovy2 dynamic with typedefs - 30ms

* groovy2 no typedefs - 500ms

So yes, while @CompileStatic groovy2 is technically approx 40-50% 'slower' in these tests, the noticeable diff for a lot of projects (especially considering file and db access will be constants regardless) will be likely negligible.

@CompileStatic is a great idea even if I prefer approach like Dart or ActionScript partial typing.

Groovy closure used dynamic scoping rules not unlike 'this' in JS which makes Closure a corner case for the static analysis performed by @CompileStatic.

Just for the sake of comparison, I ran your groovy code vs a similarly styled ruby version of fib.

Groovy 2.0.1 (simply ran with 'groovy fib.groovy') JVM 1.7.0_06_64-b24 ~520ms

Ruby 1.9.3p194 ~475ms

Obviously this is well into micro-benchmark territory but it seems even worse case is on the order of ruby's performance which is perfectly acceptable for a large number of tasks.

Is that the @compilestatic one with 520ms ?! I'd imagine not.

Certainly Ruby's acceptable for a large number of tasks, as is Groovy. I use it every day, and will enjoy the speed benefits of Groovy2 in Grails2.x in the coming months, but it's certainly production ready for a large number of tasks now.

As many people continue to point out, if Groovy's slow, write those portions in pure Java. But you can get a lot of the way there just by using explicit types when you know them. The @CompileStatic is yet another performance boost, but you can often get improved Groovy perf by typing if you want to.

520ms was for the full dynamic version.

But we did have a snafu. The 'groovy' program seems to choose its JVM in an odd way, it was using Apple's 1.6 JVM despite 'java -version' returning the 1.7 JVM.

I fixed this by setting JAVA_HOME and now get ~300ms for the full dynamic version, 22ms for explicit types and 586ms for the @CompileStatic version.

I'm guessing my method of running this is not compatible with @CompileStatic given its results. Did you run this through groovyc and package it into a jar for your tests?

This is on a 2012 MBA with the i5 cpu btw.

I actually ran through the intellij version 12 EAP "LEDA" which has Groovy 2 in it for most of the tests.

The @compilestatic stuff should work regardless, but you do need to import the compilestatic transform.

import groovy.transform.CompileStatic

Well I ran it in IntelliJ and saw the same results. Then I found the problem...

I used compile_static3.groovy from your repo for the test which contains the following:

  bench(d, "fib_dynamic", 30, "Full dynamic ")
  bench(d, "fib_integer", 30, "Groovy w/ explicit types")
  bench(d, "fib", 40, "Explicit type w/@CompileStatic")

Notice the '40' on the last line :p

Corrected results:

  Full dynamic  took 305 ms
  Groovy w/ explicit types took 22 ms
  Explicit type w/@CompileStatic took 7 ms

Which is much more like what I expected compared to Ruby at 475ms.

But.... this was in my verbal presentation - move that @compilestatic benchmark to the first test - it'll be different. There's something about running it after the other two which is adding to the benefit - caching maybe? Moving it to the first one gives me 18-20ms generally. Try it :)

Just a guess but the JIT may be detecting the recursion and replacing it with a loop. The recursion signature is the same for each method so the JIT may be able to catch that.

Letting the JIT warm up is enlightening though, I looped the 3 tests 1000 times and also looped my ruby test 1000 times and I get this:

  Explicit type w/@CompileStatic took 5 ms
  Groovy w/ explicit types took 11 ms
  Full dynamic  took 75 ms

  Ruby took 464ms

Very interesting - haven't compared it to Ruby, but the 75ms was interesting on its own. I knew dynamically Groovy would generally be faster than Ruby, but not by that much, given the runtime dynamism.

Thanks! (and thanks for the pull request fix)

Groovy 2.0 was only released a few months ago, and isn't production ready. Groovy 2.0 has two jars in it, one with static-compilation and static-inference grafted on, and the other with optimizations for JDK 7's invoke-dynamic bytecode. Programmers can't use both static-compilation and invoke-dynamic in the same source file.

Also, both jars seem to have problems. Groovy 2.0.4 was released yesterday only 2 weeks after version 2.0.2 was released "to fix static compilation and type inference problems". Grails 2.2-rc-1 (actually the first beta and not a serious "release candidate" for production use, because the releases called "beta" are really alphas) has only just been released, the first with Groovy 2.x, with only the static-compilation jar from Groovy 2.0.2 bundled, not the invoke-dynamic jar.

The facilities of the invoke-dynamic jar might never be merged into the static-compilation jar. There seems to be some infighting at SpringSource over who's in control of Groovy. Version 2.0.3 was skipped because the P.M. "compiled it on the JDK 6 instead of JDK 7 by mistake", which looks like a political play by Guillaume Laforge, SpringSource's project manager for Groovy, against Jochen Theodorou, their technical lead who's done most of the programming work on Groovy since 2004.

A year ago, Laforge hired another developer, Cedric Champeau, to put static type inference and compilation into Groovy, in order to mitigate a SpringSource business threat by Alex Tkachman's Groovy++, a static-compilation addon to Groovy which has since been abandoned. Groovy++ had to switch from supporting Groovy 1.8 to 1.7 for a while during its development because of compatibility problems in subsequent beta releases of Groovy, which may have been deliberate, and at the very least, Groovy++ served as an "inspiration" for Groovy 2.0's static-compilation jar. It's looking like Theodorou's invoke-dynamic bytecode optimizations aren't being merged into the static-compilation jar because of a power play by Laforge and Champeau who control the static-compilation jar. The project manager is diluting Theodorou's technical importance to Groovy and thus increasing his own influence.

Concerning your benchmarks of Groovy with Java, you left out Scala and Clojure, also popular JVM languages, and you chose an algorithm not used much in business. Both Scala and Clojure have lazy-evaluation in their library lambdas/functions, as will Java 8. Groovy only enables strict evaluation, which slows down many computations used in real business programming scenarios, of which fibonacci isn't one. Lazy-evaluation isn't in the pipeline for Groovy anytime soon. Its "GDK" utility methods were mostly written early on by a programmer without the aptitude for developing anything too complex, such as lambdas with lazy-evaluation. Seems like JDK 8's lambdas, due to ship in about a year, will be the first time Groovy will enable lazily-evaluated computations, long after Scala and Clojure. As for the invoke-dynamic bytecode optimizations being merged into the main jar used by Groovy 2.0 and Grails 2.2, JRuby's long stolen that show. It might never happen with Groovy.

And of course there's no spec for Groovy, unlike Scala's detailed spec or Clojure's function doc-strings. Without a spec, Groovy is still single-platform, unlike Clojure and Scala, and many other JVM languages that began their life somewhere else first.

So I don't think slow adoption by Java shops that were concerned about Groovy's performance is the reason more businesses don't adopt Groovy. It's more likely that the static-compilation code isn't production ready, just laden with problems, political infighting at SpringSource, trailing behind other languages in lazy-evaluation of lambdas (Clojure, Scala) and invoke-dynamic optimizations (JRuby), an informal-only spec that changes between point releases, the names "Groovy" and "G-Strings" evoking drug use and discrimination against female employees, and a general sense by technical managers that there's a price to be paid later down the line if they adopt Groovy.

I gave specifics - people have said to me that they weren't going to use Groovy in the past because of the speed penalty. This was year ago, well before Groovy 2 or even Groovy++ was useful. As Groovy2 continues to develop and mature, I expect more people will adopt it in their toolset - obviously adoption in Grails will be a bit boost, and that will drop in the next month or so, and continue to evolve there as well.

No one has ever said to me "the name groovy conjures up drug use - I'm not going to use it". No one I know using Groovy bothers to reference "Gstrings" except as a one-time bad joke.

As a main Groovy developer, the Jochen Theodorou, that was spoken of above, I think I have to correct the above quite a bit.

Yes, the distribution contains 2 jars now, one for the groovy parts being precompiled with the invokedynamic port as well as the invokedynamic implementation itself and another one. But! The other one is not a static version. It is the normal dynamic Groovy, that was it before. Of course you can use the indy jar and use non indy code as well as you can use @CompileStatic with it. Should there be any problems in mixing indy and static compilation, please report them, they would be bugs. In both jars indy is disabled by default and needs to be enabled on the command line or as compiler option if you want to use it. The problem with having precompiled groovy code in indy simply is, that it cannot run on a pre jdk7. The alternative would be to have no indy precompiled stuff in the distribution and we may take this path later on. So the difference between the jars is, that the second jar has additionally classes for invokedynamic support and precompiles some stuff using indy, while the non-indy jar has no indy classes and is compiled like before. Static compilation has absolutely nothing to do with it. So sorry, but you say above is large nonsense and please stop spreading non-facts and false flags. Groovy 2.0.4 was released because of some critical bugs in stub compiler, that have been a bigger problem to other projects, that want to release soon a new version. As for Groovy++. If you want to hear the full story, I will tell it you, but not in public, because I am respecting Alex, but what I would have to say about this, wouldn't sound like it and is not explained in a few lines of code. Let us just say that we had some very negative vibes between us. Him being my boss in the G2One days made things especially bad.

Then about lazy evaluation... the way I know them is for example generators. There is an article on the Groovy wiki on how to make generators using groovy.lang.Closure. I don't think that lambdas in Jdk8 will allow for example partially uncompiled code or non checked code paths until needed. So the will effectively not be better than Groovy in terms of laziness.

> Should there be any problems in mixing indy and static compilation, please report them, they would be bugs

Looks like I misread the announcements about the 2 jars in Groovy 2.0. Perhaps someone should have corrected me when I first misexplained it in my blog 3 months ago (http://groovy.codeplex.com/wikipage?title=Blog02#13).

There are, however, other issues that go back much longer. During 2006-2008 I wasn't welcomed but instead ignored then ridiculed by many in the Groovy community. Besides the public stuff, there was much more through back-channels, which appeared to be instigated by the project managers. Regarding Groovy++ and static compilation, I started building "GRegexes" atop it so had a stake in its success. So when in July last year, those project managers started a parallel project under VMware control to do the same, it looked like they were again squashing anything not under their own umbrella, despite advertising API transformation hooks allowing others to independently plug their own functionality into (Codehaus) Groovy. I started speaking out publically about their behavior then. Not wanting to give up on Groovy, I started building a version atop Clojure around December last year. Lo and behold, within a week after 8 years of neglecting JSR-241, the Groovy project manager starts writing a tool to generate a "TCK" from the Codehaus Groovy tests, from which he intends to generate a "spec" automatically. Whatever tests the current version of the Codehaus implementation of Groovy happens to pass at the time becomes the latest version of the "spec". Any changes to the "spec" will effectively be discussions on the Codehaus mailing list and Jira issues.

After all this it looks like VMware don't really want anything related to Groovy that's not under their own direct control. Other languages embrace the anarchy: Perl has CPAN, and v6 has a spec with different implementations. Ruby has gems and an ISO spec with many implementations, and Rails was built outside Matz's control afaik by 37signals. Haskell has Hackage and a spec with many implementations, but the VMware project managers seem to be discouraging independent development like Groovy++ and GRegexes, and even stonewalling regarding a spec. I have only respect for your contributions to Groovy, as well as those of Cedric, Paul, Jeremy, and others I don't know about, but I have a lot of mistrust with the non-technical overlords at VMware. Perhaps that was how I misread what they said about the two jars in Groovy 2.

Should I have corrected your misread about the announcements? Well I would have done so if I had read it. I try to concentrate on working and what goes through the lists. I don't read blogs very often. The normal channel for me is to ask on the mailing list if there are things unclear. I then try to answer them as best as I can.

Looking at nabble a bit (I did of course not look at all your messages) I don't see what you mean with being ignored or not welcomed or ridiculed in 2006-2008. as for the back-channels, that was the situation from 2003 onwards already. back then a lot happened in IRC for example. The long discussions on the list have often only been the tip of the iceberg. As for GRegexes... it is the first time I hear about it. Remember I don't usually read many blogs and you never mentioned it on the groovy lists. So I am sorry that you have this problem. I won't say Alex is the only one at fault for this. We offered him a slow integration path of his "ideas" including a rewrite of the code, because his code was quite makeshift. He was not interested without getting money. And getting new founds from VMware is a fight, especially for Groovy. If you think VMware wants to control anything Groovy, then I think I have to tell you that this is wrong. Most of VMware doesn't even know Groovy exists. This has pros (no being bothered too much by marketing) and cons (not getting much in terms of funds). The part VMware is mostly interested in, is Grails as part of their products. And they don't care too much about what Grails does exactly either, as long as it works for them. So I really don't see VMware wanting to have everything under their control. I mean you complain also about a time long before main developers have been with VMware. From my perspective the freedom we have now is more then when there was G2One (there we always had financial problems). And before G2One is was even difficult to meet. I mean when I attended the first Paris meeting I was still student and paying everything myself.

Regarding a spec... We discuss this since... I don't know, since James had the stupid idea of doing a JSR and then not working on it. Jeremy worked a lot on it, but in the end he had to stop because of potential copyright issues with Sun. We asked them if we could take their spec and adapt it for Groovy. That of course they didn't want. Then we started with a "delta" spec. But actually this cannot work for two reasons. One is that Groovy has some quite fundamental changes of the concepts, so a simple "delta" will not do it in large parts. The other is that they too change the spec over time, making a "delta" a bit difficult. Well, ok, the later is not such a big deal, if you have some persons working on this... but we never had anyone but Jeremy. As for the spec from Guillaume... What you wrote doesn't sound quite right to me. The idea is to have a document similar to the JLS for Groovy, including code. This code is to form the TCK, and "directly" executed from that spec. Quite some tests from our test suite would then go into that spec and the spec will become part of our build, to ensure we do not accidentally change the spec with our implementation. If you want to make such a project you are welcome. But then I suggest not only looking at the very small beginnings at https://github.com/glaforge/dokspek I also suggest we discuss the exact thing on the list first - and that without letting us be blinded by emotion please.

Groovy 2.0 isn't really used because it's so new. E.g. you can't use Groovy 2.0 with Grails until Grails 2.2 is released.

While I understand that its not like Groovy requires a large change to start using (unless your running Grails I guess). I just started using it a month ago when I ran into a hunk of JSF controller code that would be drastically simplified with the use of closures. It only took 1 dependency in the pom and a tweak to the maven compiler plugin and I was off and running. It integrates so easily with Java that no code changes anywhere but the controller I was writing were needed.

Hey, it's only been 17 years.

yes, for a long time, anonymous classes was enough, verbose but enough.

http://en.wikipedia.org/wiki/Turing_tarpit

More a feature tar pit, yes.

In which Java basically has taken over the world of enterprise programming. Contrary to fringe languages like Haskell and the like.

Not everything that grows like weed is necessarily awesome. Just saying.

Java forces you to write code in a pretty readable/static manner. Of course you can write 'bad' Java, but the point is that Java limits expressiveness so that YOU are replaceable/won't introduce memory issues. That's the whole point of using Java in the enterprise.

To be clear, I'm not saying anything bad about the JVM.

Kind of like how Windows basically took over the world of PC computing?

State of the Lambda December 2011 4th edition

Yes, why old news make front page? I guess not everyone is aware of it yet.

> Default methods provide a more object-oriented solution for this problem: people.sort(comparing(Person::getLastName));

Yay function pointers.

and default methods are traits but it's a secret.

Most of the effort around Lambda in the JVM has now shifted to implementing the collection libraries. You can follow along here:

http://mail.openjdk.java.net/pipermail/lambda-libs-spec-obse...

It actually looks pretty good. I can see they spend a lot of effort to make sure it fits into the existing system, keeping new syntax or new form to the minimum.

If this helps replace functors I'm all for it.

Fortunately for us, the programming language world isn't as litigious as the smartphone world.

I cannot see how to apply the lambda-calculus to smartphones…

Lamdas are not orignal.

Considering every other very high level language(gc'd) language has them(and even c++/obj-c to some extent) there is plenty of prior art.

Reply on news.ycombinator.com