Hacker News Reader

Comment by ryan_lane | Hacker News Reader

This post completely ignores one of the most important features of semver, which is dependency management. Being able to do this is really great:

some-library>=1.0.0,<2.0.0

It means I can include a library and get non-breaking changes and all security updates until the next major release without worrying that a change is going to break me randomly. It doesn't mean I don't need to do testing, but it makes it a lot more likely that I won't accidentally break in some way I didn't expect due to a change upstream.

When I want to upgrade to a breaking change release I know I need to go look at the release notes to see what the breaking change is, and then make modifications to my application.

If a breaking change is made in a minor or point release I can open a bug with the project so they can revert this change in the next minor or point release.

(edited from <=2.0.0 to <2.0.0, as this was my original intent :) )

View on HN

Possibly if you have very few dependencies, and those also have very few dependencies it will work out OK. But in a lot of circumstances, this is a recipe for shooting your foot off. For example, if you are building a server application in Node and the total number of included packages can reach into the thousands. And then if you deploy to your server and for some reason it has to update NPM, suddenly you have a potential of any and all of those thousands of packages to break, or interact poorly, or whatever. Since you have specified that anything between 1.0.0 and 2.0.0 is perfectly fine, you have absolutely no idea what secret sauce is necessary to get a working application again. So your website is down for hours while you struggle to figure out a working configuration.

/me displays closet full of T-shirts.

If you define your config to be:

    some-library>=1.0.0,<2.0.0

and your application breaks - then some-library's developer isn't following SemVer anyways, and you are in dependency hell. If some-library's developer doesn't care about compatibility, then it doesn't matter what versioning scheme you use as you will have to whip up that secret sauce anyways.

However, that your issue isn't a problem with SemVer, but a problem with NPM/node. SemVer is working as intended, but getting the "secret sauce" to get everything working can be confusing as you try and figure out the differences between "devDependencies" and "peerDependencies". If every module just managed their dependencies, it could be easier (on your development experience, but not on your hard disk and build times).

> If some-library's developer doesn't care about compatibility, then it doesn't matter what versioning scheme you use as you will have to whip up that secret sauce anyways.

It does. Semver implies semantics arbitrary versioning is not.

Sounds like an excellent argument for not using node

> you have absolutely no idea what secret sauce is necessary to get a working application again

That's what .lock files are for.

This. You define your loose dependencies, then you compile them down to a lock file that you use to test your service and you deploy with the completely locked down set of dependencies. When you want to upgrade your deps, you re-run the compilation and get updated libraries, which you test again. If you have floating deps in production you're doing it wrong.

IME this problem is particular to NPM, due to JavaScript's historically-anemic standard library which made a culture of microlibraries inevitable. The lesson is that programming languages should provide rich operations on their standard types so that users can avoid having to transitively depend on ten different versions of left-pad.

The problem might be particularly bad in NPM but it is not unique too it.

Dependency hell isn't unique to NPM, but it's also not unique to semver. The primary driving feature of semver is that it provides the framework for a social contract between library author and library user about how automatic upgrades can be performed. This works great until you the number of transitive dependencies is "in the the thousands", which I've never seen on any project save for Javascript projects.

That speaks more to the insanity of the Node ecosystem than anything else.

If you deploy unknown and untested packages to production, semver is not your problem. The utter lack of testing is. When you actually push to production, your packages should be locked to what you tested.

This doesn't prevent the use of version ranges during development or testing.

Edit: To be clear, I do not work with Node. If this is standard practice for Node shops, that's terrible.

This often results in security patches not being pushed. IME most companies are awful when it comes to maintaining dependencies, things are all to often locked until someone adds a package.

Which is why I think development should be done without locked packages, locking after testing successfully before the production release.

Not taking security patches is bad. Taking down production is also bad.

> Taking down production is also bad

I won't say it's ever good, but for most software some downtime isn't the end of the world, certainly better than having security exploits available.

I understand that view but I think that the reality is that for many customers, downtime is worse than a potential security breach. Customers would rather downtime than a definite breach, but the chance of a breach is not always as bad as definite downtime.

I'm not saying that view is necessarily good, but I think it's common. I also think it's quite common that people assume this is the client view and act accordingly.

I definitely think skipping testing with the goal of getting security patches out sooner is a terrible plan except maybe in the case of an active exploit. You can get both frequent patches and high availability.

There is also lots of Node software running in environments where downtime is unacceptable. And having uncontrolled dependency churn is a great way to break your CI and demoralize your team. I know I've spent way more time than I care to admit debugging failures in test and production because for some reason there was no subdependency freeze. It's the kind of problem that makes you very frustrated.

This discussion is based on a false dichotomy. Yes, shops can be bad at updating dependencies. That's a matter of culture. Technically, I shouldn't have to constantly scan my dependencies for updates, much less security updates. That's what CVEs and release notes are for. Not freezing the dependency tree in CI and production on account of security updates is a bad pattern.

Your problem has nothing to do with semver. The solution to your problem is to use your package manager's dependency freeze utility (npm shrinkwrap in your case), and also to select your dependencies carefully, so you don't end up with thousands of subdependencies (I know some npm packages suffer from dependency incontinence - this is a good reason not to use them).

That's a problem with Node, not with semver.

Nodejs is the first time I've encountered a deployment problem whose resolution was "you have to upgrade your package manager to a beta version".

"if you are building a server application in Node and the total number of included packages can reach into the thousands"

You know, this should be a hint that the problem is perhaps not with the versioning system.

> And then if you deploy to your server and for some reason it has to update NPM, suddenly you have a potential of any and all of those thousands of packages to break, or interact poorly, or whatever. Since you have specified that anything between 1.0.0 and 2.0.0 is perfectly fine, you have absolutely no idea what secret sauce is necessary to get a working application again.

That's not how you're supposed to use npm. You should never perform an npm update directly in production exactly because of what you describe.

The proper way to manage dependencies update in npm is to perform the dependencies' updates in your build process when building your pre-production/staging build and then use the shrinkwrap[1] command of npm to generate a file with all dependencies pinned for production. This way, for a given production deployment, you know exactly what version of which dependencies was being used and you can rollback easily if an update break something.

[1] https://docs.npmjs.com/cli/shrinkwrap

My biggest beef with semver is that it is pushed by people encouraging this concept that additions are always safe. Or the lie that people will do security and other updates on older versions.

By and large, especially in an org, if you make a change that requires another place to change, you should go ahead and make that change. Period. Thinking people can stay on the old version till they want to update is BS. They will update when they have to. And it is rarely easy, then.

I agree, SemVersion is a technical solution for what is often a human/corporate problem.

A good solution to this is to have an "edge" build that will upgrade dependencies and let you know about breakages early, but that still requires companies to acknowledge that all software needs maintaining.

The lure of technical solutions is huge. I keep calling for them, at least.

It's more of a technical tool for implementing a certain human/corporate behavior. If your organization is actually following a versioned process, then semver-aware tools are really handy; if you're note, then those tools are at best irrelevant and at worst just get in your way.

It used to work that way, though; software in the 90s and early 2000s did have sane semantic versioning, updates were backported to earlier major versions, and revision updates generally didn't break anything in production.

I bet semantic versioning is mostly championed by older veterans who have worked in maintenance or systems administration roles and are trying to get the industry to return to a versioning system that made sense once upon a time.

For my part ... while I'd dearly love to see semantic versioning come back, along with a few other good habits from a long time ago, I've given up on that and now just accept that updates will break things sometimes and there's always something to update and so something will be broken most of the time and that's just how it is so charge accordingly.

I think there used to just be less things versioned. Conway's law is probably hiding here. If there is an org to maintain it, it will be maintained. If you have more things to maintain than you do org structures, something is not getting maintenance.

I half-agree with you, because at some point, software versioning and the notion of stable apis went all post-modern (especially with open source).

But we used to pay software vendors a lot of money to care about boring things like backwards-compatibility, legacy support, and etc. Semantic versioning as a standard is a way of applying social pressure to authors to do the 'dirty work', regardless of whether they are being paid to do so. And if they don't feel like it, the result is usually "too bad for you". There's no real good answer for this that I can think of. Nobody wants to pay for leftpad.js Enterprise Edititon.

A lot of people are still doing that, including Linux distributions. Keeping earlier major versions alive and backporting security fixes is very useful.

I'm not sure I understand your argument. If someone's not going to backport a v2.1 fix to v1.6, why would they backport a v20170106-1009 fix to v201610-1005?

I'm not sure what you are saying. My argument is that it is a myth that most things will get only security updates.

That is, the version scheme States the point is to somehow be in a position to get only security updates. However, unless there is an org specifically for maintaining a library, your only hope of getting updates is to take all updates. So, to stay on the latest.

I think I must have got myself confused at some point, disregard my previous comment.

I realized too late for editing it in, but I do want to add that I do not think it is solely pushed by people encouraging this. I honestly think that most proponents of this are incredibly well intentioned.

Which just highlights, to me, that semver is ultimately good intentions masqueraded as a mechanism.

Have you got an example where pure addition breaks the existing compatibility?

> the lie that people will do security and other updates on older versions.

Semver does not say anything will be supported. You still have to know what's going on with your dependencies as far as big releases and patches go. It simply says that is old versions are still supported, they're updates will preserve compatibility.

> Have you got an example where pure addition breaks the existing compatibility?

With c/c++ additions can break binary compatibility can't they? Adding a field to a struct for instance will mean existing code that does a malloc for that struct won't allocate coorectly. I think there are a million other cases to consider too.

I don't think so. Have you got an example?

Just updated with one.

Normally in a library you'd provide functions that allocate and free structures defined by that library. That's how successful C libraries do it.

I think semver covers this pretty well actually. The public API is not just the function signatures, but basically anything that makes your API - whether it's code or documentation. So either:

- your structs in the library are opaque and the library handles all the memory management itself - addition of fields is backwards compatible, or

- your structs are open and your functions expect the structs from outside - addition of fields is not backwards compatible

The second one is still falls under "Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API."

To make a comparison in a dynamic language: you've got public function `f(d)` where `d` is some dictionary with elements `a` and `b`. Now in the next version you start to require `c` - same signature in the code, but it's not backwards compatible. It's still a public API, even if defined by the documentation.

> That's how successful C libraries do it.

What about c++? Object allocation is always handled by new on the consumer side isn't it? Also in C, does the order of function declarations matter?

I don't do C++ normally, but the situation seems similar. There's http://wiki.c2.com/?PimplIdiom for opaque objects, so you don't have to make their members part of the public API. But for things you do want to make public it's the same solution - make it a your public API by documenting it. If you know it breaks compatibility, it requires major version bump.

And no, after compilation, I don't believe the symbol order matters.

The pimpl method looks like it ads a level of indirection to every private member access. Is that an issue in practice?

Qt uses it extensively. For big, complex objects like QTextEdit it's probably the least of your concerns, but they do avoid it for simple value types like QPoint.

A good chunk of these questions I'm asking are because I half remember Qt's binary compatibility guidelines from when I did an internship at Trolltech many years ago :) A similar list is here: https://community.kde.org/Policies/Binary_Compatibility_Issu... . It looks like the virtual function restrictions are what I was talking about, that's where pure additions can break compatibility.

If you worked at Trolltech, you almost certainly know more than me. I was just a user of Qt who peeked under the covers occasionally.

It's an issue if the methods on that object are short-lived enough that the overhead of indirection is significant.

It's also an issue if very performance-sensitive routines need to read data from the object. In which case they already know exactly what's in there, so "pimpl" hiding has no benefit in the first place.

Implementations should be hidden if they are likely to change. In other words, there are are non-obvious data or method members and/or it's likely that functionality will be added in the future. These are the high-level architectural situations, not the leaf level ones where you optimize out every cyle possible.

your structs are open and your functions expect the structs from outside - addition of fields is not backwards compatible

Sometimes it's possible to handle this case (if you've planned ahead) by including unused space within the original struct defintion that you can then repurpose later.

C/C++ is hopeless for binary compatibility, under any version scheme

Qt is C++ and only breaks binary compatability on major version changes (~6 years apart).

It's hard, but certainly not hopeless.

Linux's support for ELF symbol versioning makes maintaining binary compatibility of C libraries easier than perhaps any other platform or language. It's patterned after Solaris' ELF symbol versioning, but more powerful.

glibc uses it extensively, but alas few other people know about it. Theoretically languages, like Rust, which compile to ELF executables could leverage the capability. But none do.

There are a series of posts from a libvirt developer which explain some of the mechanics:

  https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/

I personally don't bother with semantic versioning at all. Where ABI compatibility matters, I'll commit to maintaining backward ABI compatibility in perpetuity using symbol versioning.[1] And maintaining backwards API compatibility is something I try to commit to as a matter of course; if I can't then I make sure builds will break loudly.

I don't begrudge people using semantic versioning, I don't just don't think it's the best approach for most cases. It's just one of several crappy choices. But at least with C on Linux, ELF symbol versioning is a slam dunk when you're serious about the matter.

[1] I usually avoid depending on proprietary platform features. But this is the one area I'm comfortable and eager to lean on Linux.

> Adding a field to a struct for instance will mean existing code that does a malloc for that struct won't allocate coorectly.

Really? Do you ever call `malloc(4)` instead of `malloc(sizeof(int))` or `malloc(sizeof(*ptr))`?

> Have you got an example where pure addition breaks the existing compatibility?

Surely they're mutually exclusive terms. I think GP was talking about cases where the developer releases a change thinking it's just an addition, but it actually breaks compatibility.

Of course, that boils down to saying that software updates sometimes have bugs. What that has to do with semver, I have no idea.

Sorry for the very late response. I should have written it as "additions are safe." I am more reflecting on additions being more liability (commonly called "tech debt") that you have.

I think others covered the edge cases where additions literally break. The sibling that said I was likely referring to just plain bugs hits it pretty well, I think. I was not going for esoteric scenarios.

And yes, semver doesn't say anything about if a version is supported. However, it is a strong indication that it is. In a social expectations sort of way. If you don't plan on supporting old versions of what you are doing... use a new name for the new stuff. Nobody expects google guava to support apache commons, even though they are doing a lot of the same things. We do expect, however, that apache commons will make sure any bugs in old versions are dealt with.

Your biggest beef with semver is that that some people don't follow the semver spec

IIRC, Amazon deprecates old versions after some time period. That solves the "no one wants to update" problem. It also shows clearly that when a major version will not be maintained regularly.

> Or the lie that people will do security and other updates on older versions.

counterpoint: rails. At least two versions are supported at any time, sometimes older versions still receive fixes for critical security issues. Rails follows semver and they're doing fairly well. I rarely see breakages from upgrading anything but major releases.

For a smaller org, sure, especially if you have a monolithic build that would break if you made a change like that.

For a larger org, it's basically impossible to do that if you want to make forward progress.

But very much agreed that it's usually not easy to upgrade farther down the road when many things have changed. I've been trying to spread a culture of forethought and not changing things for the sake of "making things pretty", but it's an uphill battle.

> Thinking people can stay on the old version till they want to update is BS. They will update when they have to.

You can just about get away with this within an organisation, but you cannot do it to your customers if you have a competitor which doesn't impose this workload on them.

I'd like to kill off a decade-old toolchain I have to target, but people are still using the hardware it's required for. They don't want to spend the money on replacements when the current hardware still works, and why should they?

> Being able to do this is really great: some-library>=1.0.0,<2.0.0

No, no, no, no it isn't great; it's terrible.

It's makes the version used non-deterministic, and that is a recipe for build hell. The whole reason we have lockfiles, shrinkwrap, etc. is because people are specifying version ranges.

Do you ever update software on your computer?

You may lock library versions for a build pretty strictly. But the more loose is the coupling, the more elbow room you have in this regard, so you can upgrade your ngnix without breaking your wsgi apps, or your database without breaking anything — when done within a reasonable range of versions.

> Do you ever update software on your computer?

Let's just say it: a lot of web development is done by producing a blob of code and frameworks of the week, getting paid, and not caring about security and maintainability.

The poor practices and tooling are a consequence of this mindset.

Executables should pin.

Libraries should depend on ranges (though they can also pin internally for development).

You're still allowed to use lockfiles and shrinkwraps.

Anyone deploying an application to a server or distributing an application to end users should have the dependencies shrinkwrapped and the application tested with them.

Upgrading dependencies to get bug and security fixes in minor and point releases is as easy as deleting the shrinkwrap, installing the dependencies again, and making a new shrinkwrap. Then you put your application through all of its tests with the new dependencies, commit the new shrinkwrap, and then you can release/deploy a new version of the application.

Having a non-deterministic version when you build is terrible. So don't do that. Having deterministic build dependencies is also a goal, but a different goal.

Having a non-deterministic version, so you can easily upgrade all your dependencies and get the latest and greatest compatible version of everything, is great.

to defend the author here, I think they were very clear that semver may not make sense for a very specific type of CD, always-on application. The article supports the thesis as stated in text, but the title is too broad.

semver conveys information about a released product. If you're running an internal project that isn't being released to the outside world and that project isn't a library being used by other services in your environment, then you can number the project any way you want. In fact, you don't even need to number it (you can just use the git SHAs, for instance).

semver is useful even if you're releasing something that isn't a library because it can be used to provide information about backwards incompatibility in things like database schema, cache incompatibilities, configuration file incompatibilities, that'll cause a service to break if updated.

That's a fair counterpoint to the article but your example above only demonstrates the library case. You should provide an example that demonstrates the value of semver in a CD, always-on top-level application to give information about schema, cache and configuration incompatibilities.

To use the web-service example: if you present an API for others to call, even if you never version yourself, you probably still version your API. Other posts on that blog mention doing so.[1]

Well, if you release a ton of versions of your API, some of them are going to be easier to transition between than others. As soon as that occurs, you conceptually have at least major and minor versions, but in a url-based API instead of in code. Or you choose to not hint your consumers if moving from v23 to v24 is likely to be simple or not.

[1]: https://surfingthe.cloud/ideal-api-versioning/ is mentioned in the article, for example.

Ehhh... at best it argues that it's over-complicated for something that only ever has a single version running everywhere. Like a web service that you control, for which there will never be a way to request "use that old version of the code instead".[1]

Which is true. But then why even bother with YYMM.xxxx? Just use a single incrementing integer. Or none - nothing can ever go backwards, so callers are required to ignore the version in practice.

[1]: an API presenting an old external interface is something different - there's still only a single "version" of the code that's actually running.

Yeah okay. I think there is a very large group of developers who work in just such an environment: developing web applications that have a "production" version whether they are public internet or internal to companies where the previous version would literally _never_ be consumed. The reason to track a increment in that kind of environment is as a code to align feature requests to a particular release (usually by date), this is useful for all sorts of internal tracking questions like: was this feature I requested live by 4/1/2016.

I personally dislike using semvar for these instances (my own opinion only) because it feels overcomplicated as you stated. I also learned a long time ago that _any_ kind of date designator was a recipe for answering why the "April" release was actually live on June 1st ;). Recently, I've dropped back to Git SHA-1 which has alignment to a specific version of code released AND can be tracked by date and deploy tools.

So I agree with the thesis, disagree with the solution, but still think ryan_lane did not effectively refute it above.

Yeah, the blog's context isn't about libraries. And agreed, tons of developers work in environments like this.

I pretty much exclusively deal with SHAs too, since tons of tooling understands it, and there's no implied order just by looking at it - which is a good thing. A SHA might be out, but waiting to ramp up to 100%. It does tend to mean "end of (nearly) all dev work" though, which is the actually important part for day-to-day developing.

In the past I've used YYYYMMDDXX as a version number. In my experience there's no real problem with confusion about what the version number means as long as it's automatically generated so there's a clear definition. For me it was "feature merges to master, and master has passed tests". Nothing about actual releases at all, because the version of the code says nothing about what you do with it.

The version deployed is 20160215.2.142001 (on 2016-02-15, there were three deploys, with the latest build affecting them being #142001). There's your date, which is additionally free of the uncertainty whether you meant mm/dd/yyyy or dd/mm/yyyy. How is that incompatible with semver? (Nowhere does it say that the numbers have to be single digits. Integers.)

Reply on news.ycombinator.com