LInQer – C# Integrated Queries ported to Javascript | Hacker News Reader

LInQer – C# Integrated Queries ported to Javascript

github.com

147 pointsSiderite117 comments

More info here - https://siderite.dev/blog/linq-in-javascript-linqer/

Pretty neat project. LINQ is a great time saver in .NET development, I'm looking forward to messing around with it in javascript.

its a shame that linq had the overhead it does when used with collections of objects. at work right now i am often having to de-linq things

I’m interested in what performance problems you’re seeing. Maybe then I won’t fall into the same trap in the future.

Off the top of my head, the only possibility I see is allocating objects with `Select` or the like. If one can avoid that, the only overhead should be the predicate/selector method calls...?

At the very least, each LINQ method needs an enumerator, which, since it's typed as an interface, cannot be stack-allocated, even for value types (like List<T>'s enumerator). For Roslyn they noticed that the additional GC pressure was indeed a problem. But for most projects I wouldn't say it's much of a difference (and definitely don't start banning LINQ or other things unless a profiler tells you there's a problem).

Interestingly, c#’s foreach construct can operate on types that conform the the IEnumerable/IEnumerator interface even if they don’t implement it; you can have a GetEnumerator method return a struct with MoveNext and Current members and it will be stack allocated, so long as the variable on which you foreach is typed with the concrete type rather than IEnumerable.

Language integrated query comprehension is also duck-typed, so if you explicitly implemented Select and Where and friends to all return appropriately crafted structs, you could write full LINQ queries and enumerate them without heap allocations.

But you’re right that to use LINQ to objects you must have an IEnumerable because you rely on the type to bring in the extension methods.

Be interesting to imagine what c# language features would be needed to deliver a stack-allocatable LINQ...

But that’s ~one small-ish object per operator. Unless you’re using `SelectMany` or otherwise generating queries “in a loop” I don’t see how that could be so bad.

the gc pressure is normally not so bad, but there is also more indirection in each iteration. So you iterate slower, and evict more data on your cpu caches. Whether this "matters" depends of course. Are you doing a ton of work in each step through the collection? If so, the overhead of linq will be small relative to that. How much do you care about wasting microseconds and eating more battery/power? At scale these little things add up, it can get important quick. On small devices it can be important. For a personal script running on a powerful desktop, it definitely doesn't matter, and everything in between.

My LinqFaster repo has some benchmarks showing time differences for some common operations in a fairly worst case scenario (where the work done in each iteration is small) https://github.com/jackmott/LinqFaster

It would just be nice if C# reliably compiled this abstraction down to efficient code, the way C++, Rust, and Java Streams do.

There is also http://nessos.github.io/LinqOptimizer/

At least in my experience, problems with Linq are problems with optimization. For example, if you have an in-memory collection with 100,000 items, and you do collection.First(i => i.id == 123), that will be much slower than a dictionary indexed by id. (Refactor so you can do collection[123] instead of Linq.)

Some developers understand this, for other less savvy developers, it's just voodoo. In any case, if the collection is 100 items, there's no real advantage to optimizing.

Also, a foreach loop with an if statement is probably faster than a select statement, because it's not allocating a lambda and going back and forth in the stack; and a for loop indexed by a counter is probably faster than a foreach loop. Those, however, are micro-optimizations that really need their value shown in a profiler.

I once wrote a library that allowed indexing collections by different properties. It was quite intense, and it paid off as the application has lots of RAM-based queries and its internal collections can scale to 20,000 or 250,000 items.

(Unrelated: I wanted to open source the library, but my boss wouldn't let me. Sigh. Maybe I should have pushed back around my annual review that being able to put my name on a useful open-source library is important career development.)

In my experience, so much of the problems with Linq are problems with knowing the right data structures for the job, and that's something of a universal problem, Linq just makes it easier to forget to pay attention to what your data structures are; the same code written equivalently in indexed for loops or foreach loops makes the exact same mistakes. More junior programmers are just at least somewhat more familiar at spotting where they are using the wrong data type in a for loop or foreach, partly because they'd recognize when they have more loops than they expected (.First() when alone is a loop).

I've long joked that ToList() should be considered harmful, because it's an interesting crutch that gets over used and List<T> is so often the wrong data type for the operations to follow. Every time I have stepped in to performance optimize a Linq-heavy codebase, I've started by finding and removing nearly every use of ToList().

ToLookup() is one of the frequently unsung heroes of Linq that gets overlooked too often. It is the "indexing" operator, as ILookup<K, V> is the out-of-the-box multi-value Dictionary of .NET. I think ToLookup() gets overlooked often because there's no concrete, mutable Lookup<K, V> in the BCL (MultiValueDictionary<K, V> exists in the Experimental Collections NuGet package).

Deleting unnecessary ToList()s (where a query was "cut in half", presumably for debug reasons) and then converting nearly every remaining instance of ToList() to ToDictionary() or ToLookup() based on queries/usage patterns that followed was often alone a huge performance boost to Linq code that I found, simply because it was using a more useful data structure. Sometimes you don't even have to do any of the more complicated performance testing/debugging work and can stop there at the easy stuff.

So, I'm building a language with linq and the solution to this is how data is represented and if the trade off between memory and cpu is expressible via indexing. It seems that linq in c# is simply syntactic sugar rather than deploying a static analysis to extract optimizations.

A problem is Linq in C# is both: Linq against IQueryable<T> is often a case where a query planner gets involved (more often than not an SQL Database's decades optimized Query Planner); IEnumerable<T> takes the simplest in-memory generator/iterable approach that makes sense (one like most functional programming languages have used for decades).

I've thought for a long time that "coloring" IEnumerable<T> versus IQueryable<T> queries differently, at least at the syntax highlighting level, would go a long way to help people better their mental models of Linq. Especially at the complicated transition places where an IQueryable<T> query "falls back" to IEnumerable<T>.

best way to get familiar with the degree of overhead is benchmark some things with benchmarkdotnet using linq and with for loops. lists and arrays and ienumerables. you will build up an intuition for the costs involved. groupby in particular is really nasty. any time you want to use that just put things into a dictionary instead.

BenchmarkDotNet looks cool, thanks for that!

The problem is that I never had problems! (With LINQ-to-Objects performance.) That’s why I’m asking in the first place. Maybe I intuitively avoided all the pitfalls, who knows. It would be cool if you could share some (abstract) operator chains that caused problems in whatever you’re working on.

So I have some benchmarks in this repo showing some comparisons:

https://github.com/jackmott/LinqFaster

While those can look really bad - orders of magnitude slower! Keep in mind with things like Where/Select, its a fixed amount of overhead. If you are larger amounts of work, or IO in those clauses, then the fixed amount of overhead is small in comparison.

There are some common pitfalls you can avoid: Don't use GroupBy - it actually has a lot of overhead. Put things into a dictionary by hand instead. Instead of chaining together multiple Where clauses, just use one Where clause and the && operator. This saves you some overhead and has the same semantics. Consider whether you can presize a collection and then fill it up, rather than calling "ToList()" or "ToDictionary" because those will not presize the collection for you. The list will have to reallocate as it grows etc.

Keep in mind that depending on your .NET version, calls to Count() may have to actually iterate over the whole collection even if that isn't necessary. .NET Core 3 fixes some of those cases, older versions may iterate over an IEnumerable backed by a List just to get the count, when the count is already there.

At work right now, we just operate at very high scale, so in high traffic endpoints its worth it to pull microseconds of time out, and reduce memory use by bytes. Because everything gets multiplied by large numbers of requests.

Actually didn't know about `GroupByˋ. Quite strange, seeing how it builds buckets almost exactly like ˋDictionary<K, V>ˋ, according to Reference Source.

Thanks for your insight. I also found out .NET Core 3 has pre-sized versions of ˋToDictionaryˋ and possibly others.

Linq should not be used on hot paths, as it causes GC pressure. If this is what bothers your program, you can find out by profiling.

However, if you have some slow SQL query or redundant SQL queries, I bet it is performance wise better to do something about that.

In the case of Roslyn, the C# compiler, they avoid Linq on the hot paths (which run many times, for instance when typing in Visual Studio!) but outside hot paths linq is still used.

Which is annoying as hell once you are used to LINQ. I completely forgot how to write a for loop till .NET Core came up with Span<T> and friendly forced us to re-think about memory.

I hope they update LINQ in .NET to address these use cases.

Try LinqFaster which uses different techniques for better performance and less allocations:

https://github.com/jackmott/LinqFaster

i wrote that :)

I got dinged pretty heavily for using Linq in an interview test at a hedge fund several years back. I was a little miffed because they jad issues with the memory allocations and speed. Thing is, this was an ahead of time test a3md there was no explanations about expectations beyond "do X" and "do Y", etc.

Being a nonpaid test, I did what was expedient to me that met the requirements. Most of the questions were 1-liners using Linq, but that wasn't acceptable.

I probably didn't make any friends when I pointed out that I solved all of their problems and that there were no other conditions on the solution. Fuck 'em.

Interviewing with hedgefunds is a pain in the ass. Bunch of premadonnas that think their shit doesnt stink (but is probably the rankest WTF code you'll ever see) - and I say this having spent a decade at one.

A separate fund gave me a private hackerrank challenge. My code passed all of the public requirements and examples, but failed a private test case. No info given about the failure. Emailed the recruiter about my frustration with it, said fuck it, I'm not interested. Figured if theyre like that in an interview, theyre probably colossal pricks to actually work with. No surprise, I didnt get an on site interview.

Why not just use C# instead of staying in the shitty alternative? You know... https://twitter.com/knocte/status/1207544365607985152

Both these expressions return the same result in both C# and JavaScript:

0.1 + 0.1 == 0.2; //= true

0.1 + 0.2 == 0.3; //= false

It's not the language, it's https://floating-point-gui.de

I was going to reply that I thought the point of the video/tweet is that JS does choose floating point by default while others don't. But I stand corrected, C# default sucks too: it uses the double type!

And here I thought that by using F# I would even be protected from this shit but unfortunately it has the same problem... Performance should be opt-in! (favouring correctness over it) :facepalm:

Thanks

First you should state the correctness level you need. 0.2? 1/7? sqrt(2)+pi? int n=0, n<inf, sum(1/n^3)?

Look at the decimal type; it's less performant, but you'll avoid floating point issues.

https://docs.microsoft.com/en-us/dotnet/api/system.decimal?v...

Yes I know the decimal type and it's what I use in all my code, I just thought the default would be something with more precision than double/float even if not decimal.

decimal is still floating-point, just decimal floating-point instead of binary. Things like 1/3 can still not be represented exactly. It's just that for values given in base 10 there's no transformation of the representation and 0.3 remains exactly 0.3.

I'm not sure you appreciate how much performance would be lost...

This video has nothing to do with Javascript. It's the correct implementation of floating point arithmetic, and would behave the same in any language. If you want 0.1 + 0.2 == 0.3, you should use decimal, not floating point - regardless of the language.

These libraries miss the point of linq. Yes it provides a nifty syntax for munging lists, but Real C# LINQ provides that munge as a abstract syntax tree which can be interpreted before executing it.

This means it can rewrite the native language query, and say, turn it into a SQL or graphql query, or a stream filter, or check against cached data or all kinds of clever things.

It does seem to miss the "Query" part of "Language INtegrated Query". Though the methods are convenient in any case.

One of the most mind-blowing moments I can remember in my career is when I realized just how powerful the tandem of LINQ and IQueryable was.

Now, writing your own LINQ provider is no walk in the park, but is a really satisfying project and worth doing once, if only to get a deeper understanding of what's going on with expression trees and the inner workings LINQ. It's almost like a language within a language.

I enjoy LINQ, but the real power comes from when the implementation is more complex than just renaming your standard functional programming map, fold, filter, etc functions to more SQL-y aliases.

With that power comes the responsibility to actually understand what it is that you are doing, and where the abstraction leaks. If you are not careful, and do not understand the ramifications of what you are writing, it is very possible to write some horrifically inefficient code, or code that looks correct, but will break at runtime.

I have seen many a horror with Entity Framework or Linq-to-SQL when I dug in to fix badly performing code, and hooked up tracing to track the SQL code that is actually generated and executed. Multiple iteration of an IEnumerable is very easy to run into, although static analysis is getting good at flagging that with a warning. Trying to use an IEnumerable that hasn't been materialized from the expression tree into the actual data after the database connection or other resource has been disposed is another common gotcha. Understanding what operations can be translated by the expression tree engine is very important. In LINQ database ORMs, the way that foreign keys behave can vary wildly depending on how the query is written and on how the ORM happens to handle those accesses.

Honestly this just sounds more like a complaint about ORMs than LINQ in general, though I do agree with it.

I don't understand why my comment keeps getting flagged. "You need to study more" is not meant as an insult, simply as advice.

'You're too ignorant to be judging whole ecosystems as "shit".' is also not meant offensively, simply as a statement of fact.

Someone who presumably writes code for a living, but doesn't understand binary floating-point numbers, and finds themselves in a position to criticise JavaScript as being "shit" while not even having a sufficient understanding of their preferred technology to realize that it has the same (expected) behavior is objectively someone who "needs to study more" and who is "too ignorant to judge whole ecosystems as 'shit'".

Edit: Before someone says it's because I wasn't being helpful, both my comments also linked to http://0.30000000000000004.com/ which contains an explanation.

I do think that your post might help to serve some humility to OP. Knee-jerk "X is shit" posts are all too common and totally amateur. With experience you learn to appreciate that everything is just trade-offs.

That OP was so wrong about their 'smoking gun of Bad' hopefully makes them pause in the future before condemning something. Though I don't think such low-effort condemnation posts belong on HN anyways.

Yes well, it comes off as flaming, and no matter who started it it is only adding fuel to the fire. Flamewars are not productive, no matter which side of it each of the people are at.

Statement of fact or not, and insult or not, the question is, does the comment contribute to the conversation? And does it do so in a tone that is likely to be received well by others?

Even if someone is inexperienced, is telling them to study more and telling them that they are ignorant useful? Not really IMO. Aside from the thing about tone, it is also a question of whether it is giving actual actionable advice.

Actionable advice includes pointing to specific sources of information, like one of the sibling comments did.

Non-actionable advice includes telling someone to just study more in general.

> Yes well, it comes off as flaming,

Saying that someone who doesn't understand binary floating point numbers is ignorant and needs to study, while pointing them to a study resource, isn't flaming, it's an act of selflessness. It's a favor, a grant of knowledge and a nudge in the right direction.

> Even if someone is inexperienced, is telling them to study more and telling them that they are ignorant useful?

Yes

> Aside from the thing about tone, it is also a question of whether it is giving actual actionable advice.

The sibling comments did not exist when I posted my first one. It had also a link to actionable advice.

Mind you, no-one is forced to know how binary floating point operations work, as long as they don't intend on passing incorrect judgement that's brash and impolite.

If you like this syntax and work with TypeScript and Knex, you can checkout my typed-knex library: https://github.com/wwwouter/typed-knex

All feedback is welcome :)

Because there is no built in support to create expression trees from query expressions, your library is forced to re-implement common query patterns with a custom API. The result is not as clean as C#. And there is some loss of type-safety (The 'SUM' in the groupBy() example).

An alternative approach could be to parse real JS expressions using babel (at runtime) - essentially giving you the same flexibility as C#.

So you could allow users to write:

  typedKnex.query(User).where(u => u.id > 100 && u.city === "Calicut")

> Because there is no built in support to create expression trees from query expressions, your library is forced to re-implement common query patterns with a custom API.

Well, it could convert the "delegates" back to strings, and re-parse an AST from that. Not that I'm volunteering to write it.

That works. An alternative could be to do it during build with a babel based tool.

Earlier, I wrote a tool called chimpanzee[1] to parse such ASTs into SQL. Not easy (but not too hard either); there are so many different ways to express something in JS. Here's an example (using chimpanzee) which checks if an expression is doing a sort (on an array/table) and to extract sort fields and order. https://github.com/jeswin-unmaintained/isotropy-ast-analyzer...

[1] chimpanzee: https://github.com/jeswin/chimpanzee

As LINQ has been positioned as one of C#/.NET's biggest strengths their 101 LINQ Examples are a nice set of simple code examples to compare how well different languages fare against each other:

- Swift https://github.com/mythz/swift-linq-examples

- Kotlin https://github.com/mythz/kotlin-linq-examples

- Java https://github.com/mythz/java-linq-examples (Java 1.7)

- Clojure https://github.com/mythz/clojure-linq-examples

- Dart https://github.com/mythz/dart-linq-examples

- #Script LISP https://sharpscript.net/linq/restriction-operators?lang=lisp (Interactive, .NET Lisp)

- #Script Code https://sharpscript.net/linq/restriction-operators?lang=code (Interactive, .NET JS-Like)

- Elixir https://github.com/omnibs/elixir-linq-examples

- Python https://github.com/rogerwcpt/python-linq-samples

- Groovy https://gitlab.com/svkj/groovy-linq-samples

Most languages fare well in both verbosity and readability so I don't view LINQ as a major strength of C# anymore, it's just a well designed, typed query language with the USP of being able to capture and traverse an expression's AST which different LINQ providers can take advantage of by translating the intent of the query into a different DSL, most commonly used by ORMs to convert to SQL and execute the typed C# Expression logic on the RDBMS Server.

It's a nice list, but I believe there are two major faults in your comparison. The first is that LINQ supports "transducers" completely and by default for all enumerable collections. Its built in to the language and has been there from the start (almost 8 years before they were added to Clojure). Taking for example the top language from the list, Swift, as far as I know Swift doesn't support them. And my intuition is that many to most languages on the list there don't support that behavior by default. That's a fairly large difference.

The second is, if you're going to compare languages based on compactness of syntax, you can't write other languages using a compact fluent syntax, and then write c# with SQL and say the other languages are more compact.

LINQ comes with a compact fluent syntax and a verbose (but more familiar at the time and thus easier ramp) SQL syntax. The documentation when introducing concepts it used the SQL syntax. As anyone who has familiarity with the tools uses the fluent syntax. Yes you can say you used samples from the docs, but those are for teaching. You can't compare them to hand written code using a compact syntax. Use the flutent syntax for all both langs to do an accurate comparison.

I applaud the effort, but it's not an accurate comparison in it's current form.

Your entire comment suggests you didn't bother to read the project docs? These are Microsoft's MSDN C# 101 LINQ Examples, I did not write the C# LINQ Examples, Microsoft did, to show how to use LINQ. The languages are a comparison to their docs, I'm not rewriting their C# examples, otherwise it couldn't be called "C#'s 101 LINQ Examples in ....".

LINQ is the query language that's used in the 101 LINQ Examples, it's not the "compact fluent syntax" you're referring to of calling IEnumerable<T> extension methods directly, that's not the "Language-Integrated Query" that LINQ stands for, the purpose of Microsoft LINQ Examples was to showcase how to use LINQ.

The purpose of each language comparison was to achieve the same result of each example, but in the idiomatic style of that language.

> Your entire comment suggests you didn't bother to read the project docs?

Yes I both read the msdn examples docs many years ago when they came out when first learning LINQ, and clicked through and read them again in your provided links of your project docs to confirm your implementations.

I thought there were two possible things happening here. I took the charitable interpretation and assumed you may not be familiar with the fluent syntax nor the fact it is the standard syntax. Or possibly you may not be familiar with the fact that those examples are meant mainly for instructing the concepts underlying each function and are not best practices. I had all of that in mind when I wrote the following.

> The documentation when introducing concepts it used the SQL syntax. As anyone who has familiarity with the tools uses the fluent syntax. Yes you can say you used samples from the docs, but those are for teaching. You can't compare them to hand written code using a compact syntax

The other interpretation, the one I avoided was that you were not looking for a fair comparison for programmers to learn from. Your response makes it harder for me to ignore this second interpretation.

If I were the owner of a project and looking for a fair comparison, I'd be happy if someone let me know I wasn't using the correct practices in C# and would want to update my project to reflect that. Possibly creating an implementation in C# of what LINQ actually looks like in use. Instead you're digging in deeper behind the "that's what's in MSDN examples documentation".

To put it directly, if you want a fair comparison between languages for people to learn from, and make judgement from, then make it so. Don't use the antiquated SQL syntax.

Please make this a helpful project for people and use the de facto standard syntax. There are differences between languages and a fair comparison would let people learn them.

How is this too hard for you get? The examples show how different languages implement each of C#'s 101 LINQ Examples.

You shouldn't need any charitable explanation or imagination to be able to read the title and purpose of each project that's clearly explained in the docs.

Your criticism tantamounts to:

> Your 6-7 year old "C#'s 101 LINQ comparisons" series is using the actual "C#'s 101 LINQ Examples", Unfair!

Then go ahead and produce your own series, instead of wasting your time and energy on criticizing others efforts.

> How is this too hard for you get?

The fact that you appear to be getting emotional and are resorting to personal jabs means it's probably time to end this discussion.

I've tried to to explain it to you, other people in threads on this page have tried to explain it to you, the msdn documents as well themselves explain it. At this point there is nothing left to do. You're going to hold onto the "it's in msdn so it's fair" line of justification. It was good chatting with you.

Spend your efforts on reading the docs instead, as repeatedly mentioned in my comments and all docs, the purpose of the projects is to compare against C#'s 101 LINQ Examples - that's the entire purpose of the series, which is what you have an objection to, which is your problem, telling others to change the purpose of their projects to suit your personal taste is pretty naive and futile.

You're complaining that the projects does what it set out to do, which is your issue, so go ahead and create the series that does what you want - I promise I wont tell you to change it to suit my tastes.

The extension method syntax is the de facto idiomatic standard.

The idiosyncratic special syntax was misguided and fortunately is largely abandoned. Alas we are stuck with the reserved keywords forever.

The special syntax has clear readability advantages when using join's or let's or multiple from's. For simpler queries, I agree the regular method chaining syntax is clearer.

> fortunately is largely abandoned

Also fortunately, it's not totally abandoned. You can have my `let` when you pry it from my cold fingers.

They are contextual keywords and not reserved. C# has never introduced new keywords that change the meaning of existing code.

It's not a standard, they're extension methods on IEnumerable<T> in the System.Linq namespace, extension methods anyone could write and use instead.

The "idiosyncratic special syntax" you're referring to is called LINQ.

The special syntax is called "query syntax". LINQ is the whole set of features including the IEnumerable<T> extension methods, expression trees and so on.

You state that "I don't view LINQ as a major strength of C# anymore". But the examples doesn't really go into the real power of Linq, which is the combination of IEnumerable and IQueryable and expression trees.

If you have the patience to dive really deep into the Expression Tree pool, you can build some incredibly powerful tools.

See: https://docs.microsoft.com/en-us/dotnet/csharp/programming-g... http://twistedoakstudios.com/blog/Post2540_optimizing-just-i...

I'm the author of 2 such LINQ providers that rewrite expression trees for execution in different remote persistent providers:

- OrmLite https://github.com/ServiceStack/ServiceStack.OrmLite (C# POCO ORM)

- PocoDynamo https://github.com/ServiceStack/PocoDynamo (C# DynamoDB Client)

Isn't the point of LINQ to get monads into the language so that it can be used by a lot of different things (seemingly having no relations to data processing), rather than just data processing tasks?

I've no idea what you mean? LINQ is a "Language-Integrated Query" for the C# and VB.NET languages which built upon a number of features in .NET Framework v3.5 to make possible. It shipped with a "LINQ to Objects", "LINQ to SQL" and a "LINQ to XML" provider.

Of the features that differentiates it from querying functionality in other languages is its Expression Trees support that I've highlighted in my last paragraph which is essential for being able to implement alternative LINQ providers.

I'm not sure whether it's the point of it, but since the LINQ syntax is essentially just do-notation (extended by things like join, group and order) it certainly allows for usages in other monadic contexts, for example with parser combinators.

Here is an example: https://github.com/sprache/Sprache#usage

Not really. You can have monads without Linq, but there is a particular use of the 'from' syntax which can be used as a cleaner syntax for monadic operations. But this seem more like a hack than an intended use case.

It is rumored that Linq was initially implemented using monads, but that approach was abandoned due to performance reasons.

That rumour seems like it's false, because not only is "implemented using monads" a rather vague qualification, Linq has currently been implemented by requiring objects to implement the SelectMany function, which has signature:

    MyObject<B> SelectMany(MyObject<A>, Func<A, MyObject<B>>)

or perhaps more familiar to the Haskell folks:

    SelectMany :: M a -> (a -> M b) -> M b

i.e. exactly the 'bind' operator for Monads.

I think you also need to implement a 'Select :: M a -> M b' operator, rather than a 'return :: a -> M a' operator, which would be closer to a true Monad, but the 'bind' operator is the one that really matters anyway.

The rumor was that the standard extension methods on IEnumerable<T> like Join, Select, Where, Group were all implemented on top of SelectMany.

I call it a rumor because I cannot remember where I read it, so it is quite possible I have misunderstood it.

It’d be interested to see the Clojure examples redone with transducers and/or https://github.com/redplanetlabs/specter

PR's are welcome!

I wrote the Clojure examples 6 years ago as a beginner whilst I was learning Clojure but I thought its built in functions were already well suited for LINQ examples.

Do you have an example where transducers would help with verbosity or readability?

I haven't dealt with LINQ lately, but IIRC there was no way to do early termination on a sequence.

  // Not ideal
  list.Select(x => ExpensiveFunc(x)).Take(5);

Transducers can be terminated early - https://dev.to/greencoder/build-your-own-transducer-and-impr...

IEnumerable implicitly offers early termination. LINQ expressions are always implicit by default, so .Select(...).Take(n) will stop after taking N items. It's just like a foreach loop (same interface/API).

For the same reason, if you 'foreach' over the result of a linq query multiple times, it will evaluate the whole sequence again, which often isn't what you want - so people tend to call .ToArray() or .ToList() at the end to fully evaluate it once.

> IEnumerable implicitly offers early termination. LINQ expressions are always implicit by default, so .Select(...).Take(n) will stop after taking N items.

My bad, indeed it does! (Embarassing, my job used to be working on LINQ-based ORM libs. Though more than a decade back.)

Clojure's normal transducerless lazy sequences are fine for early-terminating sequences.. mostly. They're fine as long as you are ok with realizing up to 31 extra items.

http://clojure-doc.org/articles/language/laziness.html#lazy-...

I need to think about it, but they are more generic because they don’t make assumptions about the input sequence type and/or the resulting type. They are also more efficient because they don’t create any intermediate sequences.

I briefly looked at Swift. C# examples can be just as short if you use the underlying .select, .where etc. instead of the Linq DSL

If I'm not mistaken (I didn't worked in Swift for a while) Swift doesn't have proper generators. For each projection or filtering Swift will create a new array, while C#/Linq will use generators to make one really long state machine that saves memory.

I would be remiss if I didn't mention something from clojurescript land that accomplishes something similar: https://github.com/tonsky/datascript

Datascript allows you to write datalog queries in ClojureScript(and JavaScript) that operate on a b+ tree maintained by the library. It includes first class relationships and unique keys.

It looks pretty good. However, I find it similar to what you can do with wu.js or immutable.js@4, although of course LInQer would be more appealing to someone coming from C#. I wish you good luck with this project.

Thought it also worth mentioning the Julia package, Query.jl, which supports LINQ-like syntax:

https://github.com/queryverse/Query.jl/blob/master/README.md

http://www.queryverse.org/Query.jl/stable/linqquerycommands/

A big plus over previous attempts seems to be that this is written in TypeScript.

Previous HN discussion on LazyLinq, which is another JS implementation - https://news.ycombinator.com/item?id=15913571

Here's another one that decided to use JS naming conventions while providing Linq API: https://github.com/tomi/fromfrom

When would you really need this complex library over the native methods supported by Array?

I know there are overhead to convert map and set into array but it should be fine for non huge data set.

Interesting question. I share this train of thought. When I restarted writing JavaScript (thank god with TypeScript ;)) and I realized filter/map/reduce array methods, my fear of lacking LINQ vanished away. Reduce is painful in the beginning (because it is different to SQL's GroupBy) but it is a foundation for so much more (once you REALLY understand it).

Some more helpers are useful (first, last, max, take, skip ... all with variation). They should added to the core language.

Linq has Reduce as well, known as Aggregate (foldr/foldl in most functional languages).

I know, but thanks to convince function like groupby, max, etc... You never need that.

Kind of, except folds can be used to replace a lot of complex arbitrary iteration too. I hardly ever use loops anymore thanks to folds.

I wrote as a fun little learning experiment linq.js[1], which is 90% renamed wrappers for the array prototype methods, then some logic thrown in for things like any, all, first/single[OrDefault], and orderBy. It is just under 90 lines of code.

1: https://github.com/jeremyaboyd/linq.js

As learning experiment, you may try to add .d.ts or rewrite in Typescript ;)

See also IxJS, which is a pull-based version of RxJS:

https://github.com/ReactiveX/IxJS

Really cool, but the description of the project states,

"ported to javascript for performance"

In what alternative universe are things ported to javascript for performance?

You have the wrong end of the stick. The author means that he's ported it to achieve better performance than using Javascript's built-in map/filter./etc. operators because those do not lazy-evaluate and do not chain together efficiently, creating an array at each invocation.

I think word "performance" is not proper here. Maybe productivity.

I haven't looked into it yet but I see quite a few projects thinking they can outsmart the big boys by tuning for either extremely specific use cases or very small benchmarks.

The bigger argument against JavaScript in my view is the way the language is so antithetical to every lesson you'd have thought the industry has learned - even Go is safer than JS and Go is designed like it's from the 70s

Public Service Announcement: LINQ is not just syntactic sugar for processing collections.

The beauty and power of Linq is that it is translated to expression trees at runtime, can be interpreted by a provider, transformed to basically anything - sql, MongoQuery, intermediate language, etc. - and you can pass around types of:

  Expression<Func<T,bool>>

to different providers.

   repo.Find<Customer>(c => c.Age > 50 && c.Sex == “M”);

It's a compile time, not a runtime feature.

It can't be implemented at runtime, if your method is passed an opaque compiled delegate, it's not possible for the .NET Runtime to deconstruct its pre-compiled AST.

If your method accepts a `Expression<Func<T,bool>>` the C# compiler passes it the AST of the Expression instead of the compiled delegate.

For an example, paste the code below to https://sharplab.io

    using System;
    public class A {
        A() => M<int>(x => x == 1);
        public void M<T>(Func<T,bool> expr) {}
    }
    public class B {
        B() => M<int>(x => x == 1);
        public void M<T>(System.Linq.Expressions.Expression<Func<T,bool>> expr) {}
    }

and look at what the compiler generates for class A vs class B's constructors.

but, Expression can be created from string, doesn't it?

Parsed by who, the C# compiler? I'm not aware of anything that generates C#'s Expression AST from a string, other than a C# Compiler.

You can use Roslyn in code, but that's using the C# compiler in process.

The code->AST is done of course at compile time (there is no reason to delay), but I believe the expression tree itself is processed by linq at runtime. You can build expressions programmatically at runtime as well (and then ask the runtime to optmize them to bytecode). I remember doing it years ago, it was surprisingly simple (and very powerful).

Of course the Expression's AST generated by the C# compiler is analyzed at runtime, how else could it be done?

> The beauty and power of Linq is that it is translated to expression trees at runtime

It's compiled into Expression Trees at compile-time (if it's an IQueryable). If it's an IEnumerable, Expression Trees are never created in the first place (you get a function).

There is a reason I wrapped everything in Expression<> to avoid the mistake I made way too often when sending a Func<T,bool> to an EF context....

> There is a reason I wrapped everything in Expression<> to avoid the mistake I made way too often when sending a Func<T,bool> to an EF context.

Sorry, I did not follow.

Nothing will stop you from passing in a Func<T,bool> type where you should be using an expression, but if you pass it to a Where on an EF context it will be run on the client instead of being converted to sql.

I was saying that Expression Trees are created at "compile-time", and not at runtime. Here's the CIL that gets generated in the case of IQueryable - https://stackoverflow.com/questions/3894598/how-is-linq-comp...

There is an entire API for building up dynamic expression trees and compiling them at runtime.

...and zero for the inverse, which is why Expression Trees are generated by the C# compiler at compile time, not decompiled at runtime.

You are right (as far as built in support goes), but it actually could be a fun project to take IL and build up C# expression trees from it. Especially given there are plenty of IL to C# de-compilation libraries out there.

In EF Core 3.x it'll now throw an exception at runtime.

Does it AST? Time for a TypeORM 2.0; Entity TypeWork?

Is there a good high level explanation for what this is for someone not already familiar with LInQer or C#?

I'm googling around and still kinda not sure.

It's a bunch of functions that can be performed on an enumerable (array, set, , etc) type. Commonly several are changed together.

Select (aka map)

Where (aka filter)

SelectMany (aka Flatmap)

Accumulate (aka fold)

First (with optional predicate function)

Max

Min

GroupBy

Count

And so on. The readme lists them all

It's more than that though. It's also a special monad like syntax that takes advantage of those methods as well as transformation that takes the declarative form of those methods and converts them into something else ala SQL.

IIRC only SelectMany meets the criteria of a monad.

The powerful part about LINQ is it builds an AST and there are many different LINQ providers which can translate that AST into some instructions to interact with other systems. The typical one is converting it to SQL commands but you could plug anything into it.

Does this project do any of that?

The pipeline operator and some free functions is a superior alternative.

    const map = f => xs => ({
      [Symbol.iterator]: function * () {
        for (const x of xs) {
          yield f(x);
        }
      }
    });

    xs |> map(x => x * 2);

Newb question: Can this translate to a MySQL query?

Did I miss the benchmarks somewhere?

You can do a lot with pure js map, filter and apply:

https://medium.com/@aikeru/a-c-linq-to-javascript-translatio...

Reply on news.ycombinator.com