Building .NET projects is a world of pain and here's how we should solve it | Hacker News Reader

Building .NET projects is a world of pain and here's how we should solve it

blog.maartenballiauw.be

41 pointsmaartenba48 comments

Building .Net projects cannot really be solved in a way that compares well with nix systems.

- Windows had no programmability until recently with Powershell. On Linux, I can script literally everything from database installs to component updates.

- Powershell. Unfortunately they got their design wrong. Unix is a text and file based OS; you can script any server configuration with tools like awk. Windows apps doesn't have such a concept. To do stuff, you call... duh.. API!*

- The above resulted in millions of programmers not really getting it. Need a component, alright an MSI file. They don't understand how programmability of the environment can vastly simplify the act of programming itself.

- NuGet solves only a part of the problem. For non-trivial projects, there are tons of things that you need to automate besides library references.

In short, Windows is horrible if you have used anything else. And that is by design gone wrong, somewhere in the 90s.

* When they made the decision to build Powershell, they also had a chance to adopt bash or cygwin. I consider this a historic loss; the lack of a familiar/tasteful shell effectively shut (many of) the best programmers out of the Windows eco-system forever.

I am not exactly Windows biggest fan, but one of my pet peeves is talking to a Unix developer who assumes that Windows doesn't do something because it doesn't do it how Unix does it, or who blindly assume that the Unix version is superior by definition of being Unix.

Windows has had programmability for a long time. In Unix, programmability means programs that eat text and generate text. On Windows, it means COM objects. Specifically, most programs--everything from big, end-user focused ones, like Word and Excel, to server components like IIS, to low-level stuff like the Ethernet configuration--is exposed through COM objects. While this was consumable from C++ and Visual Basic since Windows 3.1, Microsoft provides the Windows Script Host, or WSH, since Windows 98 and Windows 2000 (with optional installs back to Windows NT 4 and Windows 95) that allows trivially scripting them using much simpler languages. Specifically, two languages come by default (JScript and VBScript), and plugins exist for every traditional Unix language I can think of (Tcl, Perl, Python, Ruby, and even PHP). You can also use any language that has COM bindings (most of them) to use these interfaces if you don't like what WSH provides.

Is this different from Unix? Yes. Is it worse than Unix? That depends. You can have strongly typed rich data, which is a big improvement on text streams for bigger workloads. You can trivially interact with running applications, which is something that is intermittently available on Linux through DBus, the use of SIGUSR, and the like (program-dependent). You can easily work with data that are not trivially represented by text, like nested dictionaries. On the flip side, I do understand that COM and DCOM are more complicated than text streams, but I think that's a trade-off, not an immediately gimme.

Now, the one thing annoying here is that WSH is based on COM, whereas most modern Windows is based on .NET. .NET allows a much richer object system than does COM, so Microsoft replaced WSH with PowerShell so that users could still use a common scripting language, but easily work with the .NET type system. bash/Cygwin would not have been an appropriate choice here. Modeling PowerShell on those systems was; if you look at the language, you'll find it takes heavy inspiration from traditional shell scripting languages, including bash and Perl, but it differs again due to the focus on objects v. text.

So no, it is not fair to say that Windows is not programmable. I also don't think it's obvious that forcing bash to play nice with Windows would've been better than PowerShell, which would've either resulted in giving up a lot of Windows programmability, or forking bash to work better with the Windows way of doing things--neither an improvement.

Even before PowerShell, Windows had thousands of COM objects in the system that can invoked within a scripting language like .vbs or any language that has APIs for accessing com objects like ActiveState perl. The Windows Explorer shell, Windows Media Player, not to mention Microsoft Office applications, have object models that are exposed and available to programmed against. I am not so sure that UNIX has anything better, since it seems all the functionality that UNIX has is also available in Windows. Then, there is also WMI (Windows Management Instrumentation) objects for all the low-level system operations.

I'd just like to add to this that *nix developers don't appreciate COM for what it really added to windows.

In Linux you only generally call C functions from higher level languages (and sometimes C++). Python programmers use python libraries and sometimes C libraries. Ruby programmers use ruby libraries and sometimes C libraries.

Never will you see someone calling in to a Ruby library from a Python program.

On windows via COM you get Perl calling in to C++ calling in to C# calling in to VB.NET.

So where is the problem? Installing stuff? Yeah, it's not as easy, as for example `sudo apt-get install make autoconf automake libgif-dev libtool g++ gettext libglib2.0-dev libpng12-dev libfontconfig1-dev mono-gmcs git libx11-dev libexif-dev libjpeg-dev libpng-dev libtiff-dev`.

But it doesn't change the fact that everything needs gazillion of other dependencies. I have no problem installing MSI files on production server and then leave it be for couple of years.

I think you can install via script just fine these days.

http://docs.nuget.org/docs/reference/package-manager-console...

>Unix is a text and file based OS; you can script any server configuration with tools like awk. Windows apps doesn't have such a concept.

Why is this a problem? Why would I want to use awk when I can use the powershell object pipeline? Do you even know how powershell works?

One problem with powershell is that it's ridiculously slow compared to bash, and the utilities it calls are often extremely heavy compared to unix variants.

Bash (etc) work by passing things around between processes - and there can be many of these processes in a complex shell script, possibly hundreds in parallel, or many more after each other (ever done an `xargs -n 1`?) Of course, that's only ever going to work if each process is fairly light.

As a world-encompassing scripting tool, powershell is also process based. However, it's also .NET based, and many of the sub-processes (such as new powershells or "small" scripts) are themselves implemented in .NET. However, .NET is not a lightweight VM, and many process allocate quite a bit of memory. No OS deals graciously with out-of-memory, but windows is particularly bad - allocated but unused memory must be pagefile backed, so heavy processes cause swap grinding pretty easily. (Linux tries to pretend it has memory until it's written, then kills random stuff).

Another fairly specific problem is that the .NET Regex engine is terrifying slow compared to grep; it uses a backtracking potentially exponential algorithm. Searching and extracting text from files (e.g. logfiles) is a fairly common shell task, so this is a problem. In all fairness, if you're not generating regexes programmatically, and you know the pitfalls, and you can spend time tuning problematic regexes, it's not too hard to work around (but that's still a hassle and a waste of time).

Both of these problems have bitten me in practice. I've had machines start swapping so badly that before you really notice what's going on the mouse cursor freezes (when the OS updates the mouse cursor once a minute, you know you're screwed). I've also have real out-of-memory moments with sudden lockups+bluescreens. Similarly, I've had log-parses take hopelessly long due to the poor regex implementation.

For simple orchestration, powershell isn't too bad, but that's just a subset of what shell-scripting is used for in unix.

Personally, I've got a few CLR helpers and just write scripts in F# or C# now. The regex implementation is no better, of course, but with a "real" language you tend not to rely on regexes quite as heavily since you've got easier access to other tools. And as to performance - if your subscripts are largely .NET themselves, you can stay inside one instance of the CLR the overhead of another thread/task is negligible.

The best programmers are the ones able to deliver what the customer wants regardless of the required tooling.

It's just a different bag of hurt but no more so than node or ruby. Setting up ruby boxes makes me homicidal, whereas .net feels like nothing. But then I have much more experience with the latter, and 'evolved' with it and it's precursors.

That said, building .net is relatively trivial and it's tools are caught up.

I think what he's looking for is http://fsharp.github.io/FAKE/

After 7 years of setting up ruby servers, I can't say it's ever made me homicidal. RVM and rbenv on multiple different distributions have generally all been fine. What did your distro/environment look like?

>Let’s look at the Node.js community and how they manage to do things

Node.js? The one that pulls in like 10,000 files to run a simple grunt build?

It's not perfect, but at least you don't have to manually install 10,000 files just to be able to build in the first place. Or worry that your build server is running on the wrong version of Windows. Or even do anything special to get it to build. It's not perfect, but it's better than what we have now by a huge margin.

The benefit of npm is that everything is modular and patchable. To me it outweighs the 10,000 files issue.

I said "most" projects :-) But even with that I don't have to go through my build logs 50 times before finding out which dependencies have to be installed for the build to work. The build should "just work".

This is something Microsoft needs to push instead of the insane MSI stuff. Microsoft really needs an officially sanctioned dependency manager, maybe NuGet. Something with a standardized package format based on _archives_, not _executable installers_.

The Microsoft ecosystem is light years behind OSS in this regard. Heck, CPAN was designed in 1993 and became available in 1997. Truly light years - CPAN predates the dot-com boom.

The good CPAN times :-) It was a bit of a cumbersome thing to work with as a newbie back then but at least it handled dependencies very well.

In many ways CPAN is still state of the art. It has first-class support for namespacing and mirroring, which appears to still elude most environments' packaging systems.

And I'm saying this as someone that dislikes Perl a lot.

I did some research on CPAN - comparing it to Ruby gems and Python's various packaging solutions...

CPAN is the defacto Perl standard, mirroring and testing built in. With perlbrew, localLib and cpanm, the tooling for Perl is world class. I think Ruby and Python are still catching up here. No doubt they will, but CPAN is an amazing feat of engineering.

NuGet gets the job done admirably, but it was simply introduced too late; nearly 10 years after the introduction of .NET.

It's good to see NuGet has gained traction very quickly, but there are still plenty of projects predating NuGet that have solved the dependency management problem in their own unique way. Migrating all of those to NuGet takes time and willing maintainers.

NuGet itself leaves a lot to be desired.

For one, it's used to install all kinds of crap (like jQuery) and sometimes has all too tight dependency on Visual Studio in its .ps1 scripts.

Next, all the mess with having assemblies for different versions of .NET Framework inside single nupkg -- I want to see the person who decided this will be a good idea.

Third, why include version numbers in directory names? To make it harder to update dependencies?

Fourth, dependencies. "[1.3.2,1.5)" - how cool is this?

Fifth, the whole "Package Restore" concept is flawed.

Why would installing something like jQuery be a bad fit for Nuget? It's a dependency like any other, just because it's in JS doesn't mean you shouldn't handle it the same way you would other packages.

Versions in directory names make sense if you've got multiple projects depending on different versions of the same package. Upgrading everything at once isn't always an option.

Could you elaborate why package restore is flawed? And do you have an alternative that doesn't involve putting all your dependencies in source control?

Because there's a ton of package managers for JavaScript and NuGet is reinventing the wheel here, squared. Granted, very few (if any) of them work on Windows, but that's outside the scope here.

And even if we accept that it's OK for NuGet to install JS dependencies, where do we draw the line? Install Bootstrap with NuGet? FontAwesome? Should NuGet execute SQL scripts while installing Elmah? Install MSI packages as part of "install-package"?

All in all, DLLs are the most dominant type of dependencies for .NET and NuGet should laser-focus on those.

As for multiple projects within the same solution depending on different versions of packages, that's asking for trouble. And again, why default to the illogical behavior of including version numbers in directory names when this is only justified in like 0.1% of cases?

Package Restore is _very_ fragile, both in terms "what happens if NuGet server is down" and in terms of "Visual Studio has gone mad and does weird shit trying to restore packages".

And I saw you using "source control" implying that the only thing to belong there are text-only source files. What if we call it "version control"? This will magically allow us to put all the binary dependencies and live a happy life. Cheers!

There's a ton of package managers, but none that are as built into the platform as nuget is. On top of that, what advantage would there be in having multiple package managers? All you're doing is making the package management harder by splitting it up. How would I indicate my Nuget package depends on a specific version of jQuery?

The line is simple to draw: Nuget should never install anything. It's not chocolatey or aptitude. It provides versioned files, no more. That is its focus, and should remain it, no matter what those files contain.

DLLs are dominant, but definitely not the only type of dependency. Dependencies on text files, javascript, or just about anything else need to be versionable using the same system. If not, you would run into the issues mentioned above.

Multiple projects with different version dependencies might usually be a bad idea, but it's still something you need to support. Unlike some other systems, Nuget can't be opinionated about how you manage your projects. If it was, it would be dropped like a rock by enterprises that don't want to adapt to the package manager of choice. If that makes it harder to do something the platform does for you, nobody really loses.

Package restore is more fragile than local assemblies, yes. We've seen that a few times lately when Nuget was down. These are risks that can be mitigated though, by providing mirrors and a decent local cache.

Putting any sort of binary dependency in a version control system is a problem. They can't be compared to previous versions. Unless you make folders for each version of the package you depend on, you also can't depend on different versions in your projects. And worst of all, if a dev decides to replace a package, it could break something else. Those (and others) are exactly the reasons package managers were created in the first place.

Instead of node I would say look at Maven, I never had a problem with a java project that used maven, all I ever needed was maven, and a JDK.

It may not be the best out there but to me seems pretty good considering it is as simple as download source code, and run "mvn package".

IMO maven in nowhere near in comparison with how npm works with modules.

If you have a couple minutes, take a look inside any of the Microsoft-shipped .targets file to see the mess in its full glory. That's programming in XML, kids.

It boggles my mind why Microsoft went the XML route when designing MSBuild (apparently they were copying NAnt and various other "enterprise-grade" build systems from the Java world), but it would be _so much_ better if they created a special DSL for this particular purpose.

When I see this, I want to kill a puppy or two:

   Condition="'%(Identity)' != '@(SelectedFiles)'"

Considering there's a move in the Java world to Gradle, a DSL approach, yeah... MS could have jumped to the head of the class by avoiding this XML years ago and put more forward-thinking effort in this area.

But... this isn't a new criticism of MS. In most areas, they simply adapt to what the prevailing trend is.

I completely agree. Terrible stuff.

It gets even worse with TFS and its new workflow system. In fact workflow in general is terrible and once again it's ' programming in XML'. Do MS dogfood this stuff at all?

Not that I'm really against this article, but psake (https://github.com/psake/psake) is a more realistic end-goal to handle the kind of flexibility you really need in projects. (It's accomplishable through MSBuild pre/post actions, but those are murder to edit anywhere but Visual Studio.) The problem with it, which is potentially a big one, is that psake uses PowerShell, and is therefore Windows-only. I'm hopeful that PowerShell can be ported to Unix soon as part of the general open-sourcing of Microsoft's .NET stack. I wouldn't use PowerShell as a replacement for bash, but as a cross-platform equivalent to Groovy/CRaSH, it'd be very handy for exactly this kind of thing. (Think of how Gradle fits into the Java world.)

If you want to run on both Windows and UNIX-style systems, try FAKE (https://github.com/fsharp/FAKE). It's rapidly becoming a popular choice for building all sorts of .NET projects. Scott Hanselman recently covered it on his blog: http://www.hanselman.com/blog/ExploringFAKEAnFBuildSystemFor...

Or use Rake with Albacore, a Windows/MSBuild plugin for Rake, and be done with it.

The biggest offender for me, besides nuget being pretty much useless for real work, is that you need to install Visual Studio to automate headless ASP.NET builds! That's insane. Even the simplest most barebones ASP project requires it. As a result, our Chef scripts include downloading and installing a bunch of MSIs from S3. I would love to be proven wrong.

> Let’s look at the Node.js community and how they manage to do things.

I think NuGet is much better then npm. When you download something from NuGet you get a dll file that is injected in your References. When you download something with npm --save-dev you get 1000+ files.

Just sayin'

http://earlyandoften.wordpress.com/2011/12/17/dependency-man...

I'm not really sure what point you're trying to make. This seems to be exactly where MS are moving to with nuget.

Like all the new stuff, the WebAPIs, EF 6, OData, etc. is now being published as NuGet packages instead of installers or SDKs or whatever.

NuGet seems to be a direct response to this very issue and you constantly mention it.

So what are you asking? After all they can't change the past!

The point being made here is that although many new stuff is being shipped through NuGet, there are still way to many "implicit" dependencies to the stuff that gets shipped through SDKs, installers, vsix, etc.

Setting up a build server is a good indicator really of all these "implicit" dependencies.

File > New C# Project results in a csproj file containing references to MSBuild targets and binaries that supposedly are already installed on the system. While that's a valid assumption on a developer's machine, these dependencies are declared "implicitly". Why not make them explicit and reference the appropriate MSBuild targets and libs from .NET NuGet packages? (really whish .NET gets componentized and shipped through NuGet one day). NuGet package restore could then simply pull in all dependencies required to build the software.

That is not entirely true. For example the Windows Azure SDK has to be installed. Database tools have to be installed if you want to be able to compile a database project. There are many of the new things that do not follow this approach (hopefully: yet).

Added a P.S. to the post to reflect this: P.S.: A lot of the new packages like ASP.NET MVC and WebApi, the OData packages and such are being shipped as NuGet packages which is awesome. The ones that I am missing are those that require additional build targets that are typically shiipped in SDK's. Examples are the Windows Azure SDK, database tools and targets, ... I would like those to come aboard the NuGet train!

There's nothing technical that would prevent nuget from including MSBuild targets that are then picked up after installing the package, right? (Worst case would be that you'd have to use a wrapper project which gets the dependencies and then builds the actual project, in case project includes are handled before the nuget target.)

True that! And I hope more SDK's are shipped in NuGet packages so we can define explicit dependencies instead of implicit ones.

It doesn't seem to have been widely publicized, but as of NuGet 2.5 it's been possible to include custom .targets files in NuGet packages which are automatically pulled into a project when the package is referenced: http://docs.nuget.org/docs/reference/support-for-native-proj...

Microsoft is increasingly doing it with their frameworks and components, but not for real SDKs. Windows 8 and Phone development still requires an installer to build.

It's not just Microsoft though. Other vendors need to jump on the same bandwagon. Right now, almost nobody really does, definitely nobody who sells to the enterprise market.

I think they just need to separate out the tools from the libraries, within the SDKs, i.e. all libraries through NuGet, everything else can still be an Msi, as a dev will need things like emulators and VS tooling add-ins, but none of this is needed for build server just to build the project

I whole-heartedly agree. It does create some danger there of mismatching versions between the tooling and the nuget packages. What do we do if the user has upgraded to a newer version of the package, but hasn't updated his emulator yet? Vice versa is somewhat easier to enforce, but either way it will require careful checking and notifying of the devs.

https://www.nuget.org/packages/AWSSDK/

https://www.nuget.org/packages/Box.V2/

https://www.nuget.org/packages/DeveloperForce.Common/

https://www.nuget.org/packages/Atlassian.SDK/3.0.0-beta1

Random enterprise names I picked out off the top of my head.

Okay, valid point, some already do. But that doesn't change the case that _all_ of them should be doing it whenever possible.

You're right, we need to fix the sdk's as well to tackle dll hell.

Reply on news.ycombinator.com