Show HN: LINQ for Go | Hacker News Reader

Show HN: LINQ for Go

github.com

84 pointsaabalkan51 comments

In LINQ, the extension methods defined on IEnumerable are useful, but not that interesting, because they are standard functionality in most modern languages. I also think the naming used (e.g. Where, Select) is unfortunate and it would be better to go with the naming used by other languages/frameworks that are much more standard, like filter() or map(). This implementation is also missing SelectMany (e.g. flatMap or bind), which is much more interesting than Select and Where (i.e. you can implement Select and Where in terms of SelectMany).

What makes LINQ special is IQueryable, e.g. the ability to transform a C# expression at runtime into something else, like a database query. Whenever I see article titles like "LINQ in X", I somehow know that it doesn't refer to IQueryable, which is a pity.

Anyway, these functions should be standard in any modern programming language and because Go is missing such standard functionality is one reason for why I don't like Go. Lack of generics has much to do with it, but also Go's interfaces are weaker than type-classes, combined with Google's disregard for everything we've learned about type-systems in the past 30 years (behold Dart's covariant generics) and so I do have reasons for not holding my breath waiting for a fix. Of course, Go is what many people want, so Go is what many people receive.

I would not blame Google in particular. Go team operates like they are not part of Google most of the time. You are right about the abilities of other languages to make things easier for LINQ-like scenarios. But in Go, that's as much as I could get. I don't know what else to do.

I am planning to add SelectMany, ToDictionary, GroupBy and Zip methods soon. I cannot agree you more, I also am not happy about the result I ended up with. Anyway, this was my first project in Go, I will maintain and keep adding stuff to make things easy for other people who might like to use this. But Go is 'really' not for this, I know.

Sorry for my rant on the language. It's interesting what you did and definitely helpful for other people, especially if you implement the other missing parts you mentioned.

However, most languages in common use are not "modern" by this definition (if by any) ;) I've switched from C# to Java and the clunkiness of it drives me crazy. I know that Java 8 brought a new LINQ-like API, but it won't be available for Android development for a while (not to mention there's existing codebase to maintain etc.).

This is very true -- most languages in common use are 20 or more years old and were not cutting edge when they were created. However you don't have to accept the status quo. Scala is viable on the JVM, F# on .Net, and Haskell (and increasingly Rust) if you want to avoid a VM.

C# has enough functional capabilities (for typical development needs like mine), so I never felt the need of using F#. Scala, on the other hand, looked quite neat.

The problem with Java 8 on Android is JDK 7, as it introduced invokeDynamic that will have to be supported by Dalvik, plus I think the additions to the standard library are also a work in progress. It will happen though.

For Android you can use Scala and many people do it. There's a SBT plugin for painless setup and you can work with IntelliJ IDEA as your IDE. Some people even go for Clojure. For iOS, there's RoboVM [1] and you can also use that in combination with Scala - it's very promising.

Java the language sucks, but the JVM and the ecosystem are great and you've got mature languages designed to run on top of it, languages with big communities behind them. Of course, learning a new language and finding the best stuff in a huge ecosystem where choices abound takes time and patience, but I think you'll learn to appreciate it - or if you like C# so much, there's always Xamarin, though it's a little pricey.

[1] http://www.robovm.org/

[2] http://xamarin.com/

I thought about Scala (+ Android Studio), but from what I googled it's not clear if setting it up is smooth

Or you could pay for Xamarin and avoid having to deal with Java.

If I was an entrepreneur, I might, but as an individual developer, I couldn't afford it...

It's a real pity that Google didn't go with C# as their language of choice for Android development. The world would suck less. And they actually considered it...

"If Sun doesn't want to work with us, we have two options: 1) Abandon our work and adopt MSFT CLR VM and C# language - or - 2) Do Java anyway and defend our decision, perhaps making enemies along the way"

http://www.fosspatents.com/2011/07/judge-orders-overhaul-of-...

> much more interesting than Select and Where (i.e. you can implement Select and Where in terms of SelectMany).

Well it doesn't provide a folding method either, and just about anything can be implemented using a fold.

True, but if you're into FP, then folding is for those cases in which you really need the flexibility of iterating over the collection but without the side-effects.

This has consequences. For example you can't distribute a fold over multiple threads. I'm not very keen on parallel extensions to collections, haven't found a concrete usage for them where it matters, but this does tell you something about folding - it's a low-level operation with which you're back to specifying how instead of what you want. And for collections, at least you've got the flexibility of doing a fold or a foreach painlessly, but for other container types you're out of luck.

SelectMany/flatMap/bind is the defining operation of a Monad, which is a general purpose design pattern that has many uses (e.g. Option, State, Cont, Future, Iteratee, Rx.Observable, etc). It's cool having it, because then you can painlessly filter and transform the values in those containers without pulling them out of their context. To quote Erik Meijer: "Monads are return types that guide you through the happy path".

What is a folding method?

I think we should invent a term for "LINQ, except for all the interesting parts".

Another approach using code generation is gen: http://clipperhouse.github.io/gen/

(Disclaimer, I work with the guy behind it)

Personally these sorts of libraries just strengthen my opinion that golang really needs proper generics. However, if you're already gonna use Go they're pretty neat.

Yes I know that. I have been in touch with Matt several times about this. I really look for ways to improve it, even if it is one small step at a time. Just posted a thread to golang-nuts about this: https://groups.google.com/forum/#!topic/golang-nuts/nLwWkoS7... noting Go really needs generics and maybe even lambdas some day.

Just leaving this project here. Hopefully, I will maintain it as bugs etc arises. Who knows, maybe someday Go will have these features and there will be many apps instrumenting Go for their application logic and use it.

> Go really needs ... lambdas some day

Eh? Go has lambdas:

    func(x int) int { return x * x }(5)

Indeed, but presumably they mean sugar to trim that down a bit. Like C# offers with the ... => ... syntax.

Alternate syntax doesn't seem very "Go", so I wouldn't hold my breath!

Yes that's what C# and Python offers.

From the documentation: > So if you have 1M elements as input to a plinq function, you will end up 1M goroutines.

> Theoretically, there is not much wrong with that.

Understatement in the finest tradition of man pages and RFCs.

I wrote that statement thinking about the experiments people are making using millions of goroutines and still end up low cpu utilization when goroutines mainly depend on IO. Go has a neat scheduling for that. Please correct me if I am wrong.

Just for reference, a single goroutine has a 8 kb stack size. Having one million of these things in memory at once would require 8 Gb of memory. Obviously, the scheduler kicks in, but 8 kb chunks of memory still has to be assigned and freed for every routine started and stopped.

Does it really sound alright to you? I mean it's not the end of the world or anything, but it's adding extra overhead which simply isn't needed when you're not gaining anything from it in this case.

I assume people want to use golang for the speed advantages of servicing many connections at once. If each connection is spawning hundreds of goroutines it's not going to work so well.

I gotta admit I never seen the problem that way, now I can clearly see I am wrong. Probably I need to limit number of goroutines spawned with a configurable default value. Thanks for pointing out Ryan!

Here's a gist I created a couple months ago that kind of explains the performance characteristics of goroutine per loop vs. partitioning with goroutines

https://gist.github.com/nathanwdavis/6290428

Basically, for big lists a goroutine per loop is very slow in comparison.

pretty much any task/job/thread scheduler on the planet is going to pay some absurd overhead costs to schedule and run a million tasks compared with a reasonable number (like 10, or 100), without a measurable improvement in performance. a very smart scheduler can minimize these costs but they will still be there; the state has to be stored and updated.

Not intended as a criticism- no one writes every possible feature immediately! It was me expressing enjoyment at the dry way you documented this implementation detail.

Neat. I guess Go can't do type inference on the function types though? Those anonymous function signatures are all kinds of gross.

> I guess Go can't do type inference on the function types though?

It can't, and there are no generics which is why the input has to be cast back to a Student* in every function. Also why the Sum implementation and the min, max and order (sort) situations are absolutely gross.

This is one of two main complaints I have with Go[0], which I've used professionally for the last six months or so. There have been several times when I've thought "if only I had generics..."

Instead I either have to use interfaces and casts (run-time when I would prefer compile-time) or write separate functions for each type (I wrote a Cartesian product library that abstracts the math involved to indexes and lengths, then requires a simple type-specific wrapper simply because of this silly language gap).

Despite that, I still really love Go.

[0] The other being that 1.2 randomly segfaults on my code base during compilation whereas 1.1.2 works fine. For this reason I haven't yet been able to upgrade until I determine the cause.

I wrote Go for about 6-8 months and having to do so much stuff run-time when I wanted the guarantees of compile-time started grating on me along with generics and the type system being stringly typed.

I found Haskell shortly thereafter and that is where I currently stay ;)

Well, I wouldn't say I have to do "so much" run-time, only when I'd like to write a generic library, and in that case I make a judgment call whether I want to write type-specific functions or type casts. When I absolutely have to, I just apply C polymorphism techniques.

Stringly-typed in what sense?

And the dots. Yuck.

yuck, I don't understand how go people can put up without parametric polymorphism.

http://commandcenter.blogspot.com.au/2011/12/esmereldas-imag...

"Can't imagine/understand" is a figure of speech, not literal statement to be taken by its exact meaning.

Personally, I can imagine how someone could live without $conceptname at all. It's as trivial as just never introducing that concept to to the extent they learn it. Or, maybe, that someone knows the concept but unable to use it due to insuperable force, like a lack of necessary tools for target platform(s) or corporate standard dictating what's permitted and what's not.

The point is, this is an unpleasant imagination one don't want to accept. Or something like that. But this is really getting off-topic...

I'm fully aware that it's a figure of speech. I'm calling out the hyperbole because it doesn't enlighten, and lacks self-reflection.

A much more useful comment would have been to describe how crucial $concept is to his programming practice, and how he tried working without $concept but found it a dreadful experience because of $x, $y and $z.

>A much more useful comment would have been to describe how crucial $concept is to his programming practice, and how he tried working without $concept but found it a dreadful experience because of $x, $y and $z.

Why, does anyone needs convincing of how useful Generics are?

It's 2014 already.

Well, the Go team certainly doesn't need convincing, they know how useful generics are, but you know... http://research.swtch.com/generic

Thanks for the feedback, I was being quite dismissive. In this case, at least to me, it seems quite clear that parametric polymorphism would be a win - that the addition of parametric polymorphism would allow the library to be expressed and used in a more eloquent manner.

The purpose of my comment was to draw attention to this.

Worth noting that the syntax example is slightly incorrect.

Since go auto-postpends ; to lines, you must have a trailing . on the line to allow the compiler to consider multiline statements like this as multiline statements instead of multiple (syntactically incorrect) single statements.

I presume this is just done for readability in the docs, but it's a little misleading. Correctly, it would be:

    over18Names, err := From(students).  // <--- Notice the trailing . to prevent ;
    Where(func (s T) (bool,error){
        return s.(*Student).age >= 18, nil
    }).
    Select(func (s T) (T,error){
        return s.(*Student).name, nil
    }).
    Results()

It's argued now and then that this is one of the reasons go isn't useful for chain expression libraries like this; it becomes hard to tell at a glance the difference between two consecutive statements and one chained long statement.

(The other reason typically given is it breaks error return expectations, but whatever. I've never really understood what's so hard about simple having the .Final() or .Resolve() or .Results() statement at the end of a chain to get the return code and/or error details shrug)

The real language part of LINQ isn't the simple sequence functions (.NET had them before LINQ), it's the LINQ monad embedded into the language.

While personally I don't think I've ever used the LINQ monad syntax, and wish it had been implemented as a generic feature, not every language with support for anonymous functions or closures or function pointers is "LINQ".

Some problems with the Go implementation:

Looks like go does not have something similar to extension methods, so a result() call should be put at the end to unwrap the collection.

The lambda syntax is even worst than C++ and no query expressions (aka monad syntax)

Most important: No IQueryable<T> Expression<Func<...>> so doesn't work with remote data sources (SQL, mongo,...)

But the library looks ok given the possibilities

Great comments. I have thought about creating an interface so people can just implement their own query evaluation methods and, as you said, do querying from remote data sources. That's still a long-term plan, though, I probably need to see somebody using it before digging that deep.

Does golang support reified code/expression trees/quoted code/whatever you wanna call it? That seems to be a rather critical part of making it work.

Or you use RethinkDB's RQL approach, which implements a bunch of functions that record what you did, at the cost of not being able to put relatively arbitrary expressions inside the query.

Or how else do you translate ".Where(x -> x.Age > 10)" into SQL?

Go pretty much only has extension methods.

In .NET, extension methods are functions defined on the classes you don't own, like string or System.Net.WebRequest. In Go, you cannot define functions on builtin types and exported types from packages other than your own package. So Go way of defining functions on types is not exactly extension methods.

I am afraid that's actually not 100% accurate. Microsoft also uses term "LINQ" for method-based query syntax. See here: http://msdn.microsoft.com/en-us/library/bb896339(v=vs.110).a...

Yes, I know Microsoft put "LINQ" over the entire thing, but calling map, filter, fold etc. "LINQ" seems to be a huge stretch. Method-based query options were around in .NET and C# before the LINQ name existed.

It's not really "Language INtegrated" if there's no real language aspect, is it?

Can't say I'm a fan of the syntax, but nice work nonetheless.

Nice job but sorry, that is some hideous syntax require to make LINQ Go ;-)

Was expecting the concision and elegance of LINQ-to-SQL.

Reply on news.ycombinator.com