Writing your own LINQ provider
aashishkoirala.wordpress.com
aashishkoirala.wordpress.com
Personally, however, I think C# as a language strikes the right balance in terms of power, ease-of-use, learnability and code readability. Even if you have a problem with Microsoft, I think it is wrong to let your "MS-hate" translate into "C#-hate". It is a beautiful language and the open-source community around it is growing rapidly. The progress made by platform-independent endeavors (case in point Xamarin) is admirable.
In the unlikely scenario that all of programming were to converge into one language, I would want it to be an open-source, platform-independent version of C# (with maybe just a tad more constructs borrowed from the functional programming world).
P.S. I also think it is a mistake to lump C# and Java in the same bucket. Ten or more years ago, maybe. Today, they are VERY different.
Was wondering what, in your opinion, are the largest differences between Java and C#? It's been a long time since I've done any Java, and while I can definitely name a bunch, I thought it'd be interesting to get your take on the most impactful or meaningful differences.
In terms of differences with Java (and limiting this to just the language and the syntax), LINQ is a good example of it :) I guess you could classify a lot of the other differences as synctatic sugar, but those little things add up and make a real difference in terms of code brevity, readability and developer productivity. To name a few (and I know that Java, especially with v8 has started adding support for some of these): Properties (and auto-properties), Delegates, Events, Lambdas/Anonymous Functions/First Class Functions, Extension methods, "using" for IDisposable (although Java 7 has something similar in AutoClosable), the lack of forced checked exceptions, the newly added async/await keywords, object and collection initializers and anonymous types, etc.
F# is definitely a nice language. I haven't done any large scale projects in it, but what I have done has been very nice. Check out type providers if you haven't already. They're really interesting and handy. I switched over to F# as my side/fun language from Clojure when my day-job became .NET. It's not that I'd say they're equivalent languages, but the way of thinking about most things was similar enough that it felt right. On top of that, it was nice having a fairly strong type system to back things up. It meshed more with how I think about things.
Hah, I probably should have assumed LINQ would top the list of a linq post. I remember before I knew what LINQ was about. Once I started using it, it quickly became one of my favorite parts of the language/ecosystem.
I really do feel that the things you've mentioned do make it a much cleaner and more consistent language than Java. I've got a friend who switched from full time Java to C# recently. I'll have to see what he thinks as well.
I regret I didn't knew about it when I wrote my first LINQ provider.
I used it to build a LINQ provider for a legacy mainframe database system, abstracting away the incomprehensible table and column naming, weird date systems, EBCDIC, and other nastiness involved in querying that system using straight-up SQL.
One thing I found is that implementing a custom LINQ provider gives you a better understanding of how Entity Framework runs your queries. You get to see how queries get split up into the parts executed in the CLR versus the parts that get translated, and how expressions get rearranged and rewritten.
Async/await really is a great abstraction on top of task parallelism; it's easy to understand without having to wade through a zillion blog posts explaining how it's just like burritos, Lego blocks, the OSI model, or homeobox genes, or underpants. Which means that it's something you could actually get away with using in enterprise software development, rather than something that'll earn you a reputation for developing software that your colleagues wouldn't be able to maintain.
I have to disagree to you also on the ground that almost everything in Haskell is (and many important things in Erlang are) async without any annotations. Haskell is lazy and that alone makes many operations "asynchronous", so to say. It's run-time system is heavily threaded and employs work stealing to fit millions of green threads into several system threads.
You can write your own scheduler [1] for your purposes, if you please. As fine grained as you want it to.
[1] http://hackage.haskell.org/package/meta-par
In fact, .Net is years behind of Haskell and Erlang. Just because it borrows features from Haskell (and other languages) libraries and bolts them into language.
The difference here is that things in libraries are first-class, you can create them at run-time. That does not often apply to language constructs.
thirsteh's cynicism is misplaced. node.js is effectively predicated on asynchronous implementations, and you have to work hard to do otherwise. The end result tends to be very efficient implementations that can serve large numbers with ease. .NET bolted on asynchronous sugar as it matured, and while that is very good, and you can make efficient implementations, most don't. So it's not really reasonable to dismiss the talk about one because it's possible in another.
EDIT: thirsteh is continually editing their posts so assume their commentary is very different from what it originally was.
Despite what you seem to imply in your comment, I/O does not block an entire thread in any of these languages, however the control flow in a single thread does block, i.e. err = fireTheRockets() does block until fireTheRockets completes, but performance-wise it is equivalent to fireTheRockets(function(err) { ... }). Only now you're saved the callback spaghetti.
Node.js advocates are literally peddling callback spaghetti as an advantage because "it lets you write much faster applications with ease", yet Haskell's threads and blocking semantics blow any Node.js out of the water by an order of magnitude. (Check out Mighttp/Warp or Go's net/http vs. Node.js.) I come across as cynical because that is sad. It feels like we're travelling back in time, not forward.
The only thing that makes node.js "good" when it comes to threading and async is that it doesn't require synchronization as there is no concurrency. (Same thing for Python with Twisted.) Neverminding that that opens up its own can of worms (I hope you don't do any actual computation in your program, as that's going to block everything): Pretending that it's anything more than that comes across as incredibly ignorant of contemporary language design.
And you would lose.
This conversation was about .NET, and then you (rather oddly) decided to pull in nodejs. It has nothing, at all, to do with Erlang, Go, Haskell, or any other language, and the tactic of using them as the "big brother" to discount another language is petty and disingenuous.
The reason I mentioned those languages is that there is essentially no way (short of using the C ABI) to actually do blocking I/O. It doesn't mean that you can't write performant asynchronous programs in C#.
Maybe you should knock that nodejs chip off your shoulder and get over it. Clearly you had some sort of argument with someone about it at some point, but your arguments just make you look absurd.
Here is what you said: "thirsteh's cynicism is misplaced. node.js is effectively predicated on asynchronous implementations, and you have to work hard to do otherwise. The end result tends to be very efficient implementations that can serve large numbers with ease."
This is what I responded to. Your statements are simply not true. And no, I wasn't the one who brought up node.js either.
My gripe with node.js is that it claims to be something much greater than it is, and people gobble it up. If you want to write Javascript, sure, it's a great tool. Just don't try to sell it as the only, or even a good way to write fast applications. That is what is disingenuous.
Your quote comically excludes the preceding paragraph about .NET, and the following sentences specifically contrasting .NET with nodejs. Instead you continue this transparent charade of pretending that I said something that I didn't.
This boring side discussion is done. You have said absolutely nothing of value, but should feel relieved that you took a irrelevant shot at nodejs for no particular reason.
I do, however I fear it will do very little to stem the tide of frontend developers discovering the awesome powers of asynchronous callbacks in Node.js, and continuing to repeat the meme that it is the best way/a good way to write fast applications. Every other thread on Hacker News is about Node.js these days.
C# is a great language, and there are several libraries with fantastic design aesthetic; one that is much better than most of Ruby/JS.
Also -- F# is completely open-source, so there's no "Microsoft tax" to be paid. In fact, a large portion of the F# community runs exclusively on non-Windows platforms (e.g., OS X, Linux, FreeBSD, Android).
For example its easier to get started with F# with MVC on the Xamarin IDE then it is in VS. Hope it takes off.
It's painful when you have to go into your IDE, step through a bunch of stuff, just to make a small change, and then a couple minutes goes by before you can check your work. Integrate some larger testing, and it's worse.. try to break you project up into smaller libraries (as you would with node) and it gets worse.
Don't get me wrong, I really like C#, and love the Razor view engine... but .Net and VS as a whole just feel really sluggish and bloated. With node, I can fire up an app, and include modules with relative ease.. I don't get the same level of intellisense as VS with C#, but I find I need it quite a bit less, and tend to be just as, if not more productive.
I started with C# in mid 2001 (before VS.Net RTM), and until Node.js it was by far my favorite environment.
Kind of like how Rails devs have tests that run for minutes?
I kid. Sort of. Both instances are problems of poor design and architecture. C#/VS don't really pay off until you start having larger code bases. Which, for web apps, doesn't happen...until it does, and then it sucks. The semantic analysis afforded by more static languages like C# becomes a real boon then. I'm not going to argue that types will fix the Software Problem, but they do help considerably, especially when an API shifts out from under you.
I love VS (and won't write any C++ without it), but it is pretty hefty. Some of the recent UI changes are highly questionable in my mind, but, that's MS for you.
It's about learning your tools, and how to bring in what you need, and keeping out what you don't. In the end it's almost as much about technique as the tools, but sometimes it's easier to adopt new techniques while adapting to new tooling.
Also, a lot of script-y environments discourage you from thinking about it: Rails' autoload, DHH's rants, no compilation/linking step, etc.
Now, we promote such a way in the name of productivity. Little wonder SRP gets squeezed out.
I mean, C# is a fantastic language, but a lot of the libs MS has bundled into it are absolutely horrible and people use them anyways because they're the blessed first-party product. And rather than fixing the problems, Microsoft simply creates a new replacement lib with a new approach. I mean, how many kicks at the can did they get with remoting? DB access? Serialization?
Microsoft has gotten a lot better recently, taking an open-source-style development approach to Entity Framework and MVC, but they're very late to this particular party.
.NET has been around for a long time, long enough to have gone through the rise an decline[1] of many open source alternatives:
DHTML, PHP, Python, RoR, NodeJs
And the language and framework has been thoughtfully improved over that period of time.
> Microsoft's bloated and broken enterprise tools instead of sensible, rapidly-released libs.
Developers have been writing great software with these tools for decades.
And rapidly-released libs could just as easily be classified as "not-yet-ready-for-production."
> I think the big problem is that MS
In my estimation, the biggest problem is that using .NET "requires" the rest of MS along for the ride. Yes you can use mono, yes you can use apache, but everything just works if you use the WINS stack.
I like .NET and I really like C# as a language, but I'd rather have my infrastructure be linux.
[1] Not all of these have "declined", but popularity has shifted focus.
But where other libraries would have to compete in the open market-place of ideas, Microsoft's failures are still "blessed" by MS and so we still use them.
But the bigger frustration is that they don't seem to really properly iterate on the libraries. They get fired out as part of a .NET major version release and get at most minimal bug-fixes beyond that. If there's a major flaw in the lib, it will not get fixed. Instead, a completely new library with its own flaws and a whole new learning curve will appear, and the old lib will remain and languish in its "functional but still fundamentally frustrating" state.
Like I said, I'm ecstatic to see that MS has moved away from this methodology with Entity Framework - they've de-coupled it from the .NET core and are iterating on it faster than Once Every Major Language Release.
This is definitely changing with NuGet: Immutable collections, OWIN, SignalR, etc.
I've decided for my own side projects to start using Django and Postgres.
I can be pretty much be confident that two years from now whatever version python, Django and Postgres are, my gained experience and knowledge will still be very relevant as the maintainers of these projects don't have a mandate of building a brand new API, or what have you every year.
I do not mean to raise the ire of defensive .NET programmers by saying this: as with most things, used right it is a beautiful thing, and there is nothing fundamentally wrong with LINQ. But what it was used for in practice was almost always grotesquely inefficient.
If someone had to make a loop to iterate objects to find a member on every function call, pretty soon they'll realize they should using a System.Collections.Generic.Dictionary for instance, or some other appropriate data structure (e.g. Need sorted data? Then use a sorted tree, for instance). When it's a simple LINQ query, though, it's one seemingly harmless little line, so it tends to get a pass.
Pretty soon profiles were just hundreds (if not thousands) of different cases where LINQ was used as a cure-all, destroying performance under any load at all.
LINQ is not magical. It is doing the same dirty work that you would be doing yourself if you needed to filter / sort / recast, but provides syntactical sugar to do so. The compiler and runtime will happily generate horrifically inefficient code if you request it, with no warning or complaints.
The tldr; is that it is a fantastic set abstraction that is more often than not grossly abused.
I find it incredible when I see comments like this. What you should have done is instigated a mentoring programme for the programmers who aren't yet up to speed. LINQ is one of the best things to happen to the language and you just go and kill it.
I think the real problem is Cargo Cult programming. People are using linq (especially in combination with EF) and not understanding what's actually going on. So they end up getting things like n+1 selects on collections with thousands of items. They don't understand delayed evaluation. Or how sorting effects a Where() call.
The problem is programmers who think they can take some code from stack overflow and use it without outstanding it. That problem is not unique to LINQ or .NET.
It absolutely isn't unique to it -- powerful features allow programmers to shoot themselves in the feet, and this is true since we began this profession.
When dealing with larger applications and datasets it has the potential of easily imposing enormous costs, of the sort that absolutely eclipse most other anti-patterns. Appending on an immutable string through recursive function calls suddenly looks like child's play.
The truth is that in most apps, these inefficiencies simply don't matter, and to all of the "just learn how to use it right!" replies, I would say that odds favor that those people aren't using it right at all. But in the end it just doesn't matter that much because small amounts of data, low demands, etc, just makes it an irrelevant cost. Most apps are running with an excess of CPU resources relative to the demands on the app, so it just doesn't matter.
Many .NET developers have never run Visual Studio's profiler. They don't even know what the costs are, but we're in a world where 1500ms to load a basic form is okay.
When you deal with very large scale financial data, as I do, however, your world is a very different world. You don't just have a scrum where you try to teach people that LINQ isn't magic (which of course we did). The risk factors become much higher.
LINQ isn't just about hitting a database either, it enables a more declarative / functional approach to development in general, which brings huge benefits in terms of parallelisation / robustness / testability.
It seems absurd that you would block your team from using it.
Edit: Oh and btw, you're not the only person who has to deal with "large scale data". Many of us do and manage just fine with LINQ.
And yes, those milliseconds matter. And it's very strange that I noted that after repeatedly encountering performance issues with LINQ, you say that we should have oversight. Do you understand that such is oversight? That the rules and exceptions came about because the recurring issue with LINQ (particularly LINQ to Objects) being an enormous performance issue?
It doesn't matter to you, probably because you don't even know what the cost is [and that isn't meant to be at all a slam. For most software the cost simply doesn't matter]. And that's fine. But save the continual exclamatives (which you always add right after defensively down-arrowing).
Though I have to chuckle at the notion that LINQ helps with testability, or robustness for that matter.
No, you're exaggerating your position to try and justify it. There is zero substance in what you're saying. All I see is someone who appears to have made an irrational decision about a technology because he personally doesn't like it. There's no fundamental reason why LINQ to Objects should be any slower than say a foreach over a set. You can get is a mess with either approach. If you're not prepared to mentor your team, or do code reviews to check for problematic usage then it's a fault of yours, not the technology.
> It doesn't matter to you, probably because you don't even know what the cost is [and that isn't meant to be at all a slam. For most software the cost simply doesn't matter].
Of course it's meant to be a slam, you have no knowledge of my experience of optimisation or my deeper understanding of how frameworks like LINQ works, so instead you assert that I "don't even know what the cost is". I have spent a large part of my career doing 'to the metal' optimisation, so I will ignore that comment.
> Though I have to chuckle at the notion that LINQ helps with testability, or robustness for that matter.
If you don't understand why imperative and functional code are different then you won't understand why LINQ is inherently more robust and testable.
There's no fundamental reason why LINQ to Objects should be any faster than a foreach over a set. Which is precisely the point. LINQ has an amazing way of hiding details in a manner that allows for massively inefficient code to look completely innocuous. For throw-away, one-off "command line utility to convert A to B" type code, it is absolutely golden. For long-term use code that may be called millions of times, it can be disastrous.
You have no interest in a reasoned discussion, and your argument style is best described as defensive. That is all. Have a nice day.
I'm out of this thread, it's not adding to the discussion.
This, in a nutshell, is your argument. And it's an incredibly poor, out of place argument that is founded on ignorance, and does absolutely no service to .NET.
Performance in most modern applications is achieved through proper algorithms and data structures. LINQ is often a hasty short-cut around those, but it offers the illusion of elegance such that many people (such as, apparently, you. It's all functional-like and such, so it has to be good, right?), are blissfully unaware.
Again, a lot of the time that's perfectly fine and causes no real harm -- similar to the way regular expressions might be a costly but powerful catch all -- but in other cases it is not good. That is the case with large scale financial software. If you have a different experience, good for, but simply saying "incredible!" over and over again does absolutely nothing to make your case.
LINQ is really one of the best things in .NET. Just like any tool, you need to learn it before using it properly.
Believe what you want. I will say again that of the three "I can't believe it!" responses thus far, I'll bet zero have ever even run their profiler. Ignorance, as they say, is bliss, and in many cases is perfectly fine.
For example, all queries that are generated for database calls via LINQ are reviewed and analyzed to make sure its what is expected.
You are right, LINQ is not magical and was never meant to be. I don't think using LINQ is an automatic ticket to leave your brain at the door.
In your case, it's easier to blame the tech and stop using it than to take responsibility for training the team to be better developers.
I would also imagine that on this team the use of var was also restricted?
Also, check out LinqOptimizer -- it analyzes your LINQ queries and can optimize them for better performance (and it's free!): https://github.com/nessos/LinqOptimizer
You need to know how to parse function bodies and I strongly suggest algebraic types approach, e.g., using SpecificType v = genericTypedVar as SpecificType and checking if v != null. You have to write your own parametrized type supporting Where, Select, etc.
And you're good to go.
It took me three days to write first LINQ-like thing.
That said, I think for stuff that is supported, better to stay within the "standard approach".
But I get your point.