Hacker News Reader

Comment by userbinator | Hacker News Reader

Small modules are easy to reason about

Yes they are, in the same way that a book in which every page consists of a single word is easier to understand than one with more content per page.

By focusing on the small-scale complexity to such an extreme, you've managed to make the whole system much harder to understand, and understanding the big picture is vital to things like debugging and making systems which are efficient overall.

IMHO this hyperabstraction and hypermodularisation (I just made these terms up, but I think they should be used more) is a symptom of a community that has mainly abandoned real thought and replaced it with dogmatic cargo-cult adherence to "best practices" which they think will somehow magically make their software awesome if taken to extremes. It's easy to see how advice like "keep functions short" and "don't implement anything yourself" could lead to such absurdity when taken to their logical conclusions. The same mentality with "more OOP is better" is what lead to Enterprise Java.

Related article that explains this phenomenon in more detail: http://countercomplex.blogspot.ca/2014/08/the-resource-leak-...

View on HN

Perhaps another reason for this is that Javascript is inherently unsafe. By relying on these small rigorously tested libraries they are avoiding the need to test their own code, and thus avoiding basic null check or conversion errors.

Other languages handle this with compilers, may be strongly typed, and/or have features that don't allow nulls. Javascript doesn't exactly have that luxury. So maybe it makes sense in Javascript, where it wouldn't in other languages (though one could argue this is a flaw in the language design).

edit: to be clear I'm not really defending the practice, but trying to give a little perspective.

The equivalent in C would then be replacing

#include <math.h> and "-lm"

with

#include <sin.h> #include <cos.h> #include <tan.h> #include <sinh.h> …

and "-lsin -lcos -ltan -lsinh …"

Which is nuts no matter what language you are coding in.

Oh, but javascript is dynamically typed, so it doesn't matter if your sin-module works with a different kind of arbitrary-precision decimal number module than your geolocation module, your cosine module or your openg-draw-triangle or opengl-draw-circle modules... /sarcasm

It's not sarcasm if it's true!

OT but apt to this community - That's what I thought, but after a lifetime of 'correcting' people, I looked up the definition of sarcasm. Turns out that the sarcastic-sounding truths I'd been deploying for decades were indeed by-the-book sarcasm rather than 'I can't believe it's not sarcasm'.

Key quote from brilliant word-talking-guy Henry Fowler: "Sarcasm does not necessarily involve irony [but it often does] ... The essence of sarcasm is the intention of giving pain by (ironical or other) bitter words."

So, we circle back to Node.js by way of pain and bitterness. Sounds about right.

The difference with C is that with a good linker, functions you don't call are removed from the resulting binary (or are maybe not even in the binary if you are dynamically linking to the c runtime).

With javascript however, if you import a hypothetical "math.js" instead of just "sin.js", "cos.js" or "tan.js", then you'll need to download and evaluate a whole bunch of javascript that you might not need.

I'm not defending javascript, because I dislike it and avoid it where possible, but I can understand some of the reasons why short, specific modules came about.

what you describe is the "-flto" option in GCC and I believe the equivalent in JS is donr by Google Closure compiler.

He described not Link Time Optimization (-flto), but the very basic linker functionality: to only include required functions. C/C++ has a weird compilation model where each source file is translated to machine code separately, with placeholders for unknown function addresses. Thus it is trivial to take only required functions.

-flto, on the other hand, allows to reverse this process to allow interprocedural optimization across different translation units.

He's describing unused-function removal... which "-flto" is the correct option to use (on GCC). like you say, without this, removal is only performed within the compilation-unit, but my statement was quite correct.

@majewsky: what you're describing is more akin to "-ffunction-sections" and "-gc-sections" which will split ewach function into a separate section and garbage-collect those sections.

@TickleSteve below me: "without this, removal is only performed within the compilation-unit"

Not quite. When you link to a static library (.a) with lots of .o object files in it, only those object files will be linked that are actually used by your program.

I first learned about this when I looked at the source of dietlibc, and wondered about every function being in a separate file. That enables the aforementioned trivial optimization, even without -flto.

Thanks for the correction. Didn't know that.

I saw that discussion years ago about the G++ String implementation. That was a big .o file.

Webpack 2 (probably the closest thing to a JS linker) can do that same tree shaking.

> #include <sin.h> #include <cos.h> #include <tan.h> #include <sinh.h>

I've written worse - at least those cover multiple variations each (or overloads in C++ for float/double/std::complex?/...)

While I'm not a fan of the enforced java route of 1 file = 1 class, I do trend towards 1 file ~ 1 (main, public) thing - which might be a single function with no overloads. #include <string_left_pad.h>? Better than #include <everything.h>, which I see far too often...

I don't have to figure out which grouping of things my coworkers decided to throw something into if I can just #include by name - whereas aggregation headers more often than not will trigger a full project file-system search.

Unnecessary #include s don't accumulate nearly so much when you don't have to audit 100 functions to see if any of them use features from said header.

I don't trigger a rebuild of everything when the only things that #include stuff actually need it.

Lots of benefits!

> and "-lsin -lcos -ltan -lsinh …"

Or sin.o, cos.o, tan.o, sinh.o... which suddenly sounds a lot more reasonable. I wouldn't bat an eye at string_left_pad.o either.

Sure, I want to use tools to simplify the use of those micro-modules by aggregating them into libraries for convenience. But that's not a knock against micro-modules - just their packaging and tooling. Rewind time back far enough and I would've been manually specifying .o files on the command line and kvetching too...

Java doesn't enforce 1 file = 1 class but rather 1 file = 1 public class, which is exactly what you asked for. You can put as many private classes in the file as you want.

This isn't accurate either. Java does not enforce 1 file = 1 public class but rather 1 file = 1 public top-level class.

For example this is totally legit:

  // ClassA.java
  public class ClassA {

    public static class ClassA_Inner_Public_Static {

    }

    public class ClassA_Inner_Public {

    }

  }

  // ClassB.java
  public class ClassB {

    ClassA classa = new ClassA();
    ClassA_Inner_Public classA_Inner_Public = new ClassA().new ClassA_Inner_Public();
    ClassA_Inner_Public_Static classA_Inner_Public_Static = new ClassA_Inner_Public_Static();

  }

Not exactly - I didn't ask for enforcement, and the irritation usually arises from wanting the occasional secondary type - such as an enum - to be part of the public interface (thus forcing the type itself to be public, naturally) without either:

1) moving it into a separate file, for the simplest two value one liner

2) making it interior to the other class, with all the verbose lengthy names (and semantic implications) that this might entail.

I want clear and concise names, so I lean towards #1. And if an enumeration gets more shared between multiple classes, or collects enough values and comments to merit it, sure, I'll move it into it's own file. But forcing me to shove it in a separate file before I've even decided on the best name (meaning I need to rename the file repeatedly as well), and before I'm even certain I'll keep that specific enumeration, is just meaningless friction.

So have libmath depend on libsin, libcos, libtan and libsinh. People who want the kitchen-sink version can get it. People who want just one specific submodule can depend on that. What's not to like?

Having a full set of math functions isn't kitchen sink, it's the right level of granularity for a module. If I want math functions I really want them to all be written by the same author and kept in lock-step as a unit.

Why don't you want the whole module? Because it's bloat? Surely it's far less bloat than one submodule per function?

A lot of javascript is going to be deployed to a website.

Every line of code that isn't used is bytes you're sending to every user unnecessarily

> Surely it's far less bloat than one submodule per function?

How is that bloat? It's the same LoC whether it's 1000 modules or 1 after it's been compiled/minified

Bloat in the source code, build system, dependency management, overheads per submodule, complexity.

I'm not arguing that there shouldn't be a math module but that you shouldn't split it up into such (imho) crazy small parts.

Closure Compiler can remove any unused functions.

It's not bloat in the source code - if you don't care which pieces you depend on you declare a dependency on math, if you do care you list the specific pieces.

It shouldn't be (impactful) bloat or overhead in the build system or dependency management. That stuff should just handle it.

Most of the time you don't care and can just declare the dependency on math and let the closure compiler handle it. But sometimes it might be really important that a project only used cos and not sinh, in which case you want to enforce that.

Ooops. Upvoted you by mistake. Why would it be so important?

Maybe you're running on an embedded processor that can do sin/cos/tan with high performance but sinh performance is bad. Maybe part of your project needs to depend on an old version of sinh with a correctness bug so that you can reproduce existing calculations, so you don't want any other parts of the code depending on sinh. Maybe your legal team doesn't want you using sinh because they believe the implementation might infringe some patent.

If basic, bog-standard functionality was built into the standard library, then it's not bloat that you have to deal with. 500 kb or a meg or two for a decent, full-featured standard javascript library isn't going to even make a dent in the install sizes of web browsers, even on awful mobile devices.

In the compiled artifact there might not be a difference. But developers tend to ignore the issues outside their scope. So yes, for a developer it might not make much difference. For people who need to make sure that you can compile and deploy, it's much more complexity.

> What's not to like?

That depends; in the case of mathematics libraries they have to be bundled together because of functional dependencies. It's either that or ignoring DRY principles -- which if you do that, you're ignoring the entire point of what breaking things into submodules is intended to do.

Separate interface and implementation. cos and sin might well use common underlying code (by depending on the same module), but they should expose orthogonal interfaces.

There's a cognitive load to every additional thing you need to know to use a module. At a certain point, added flexibility becomes more trouble than it's worth.

But there is no cognitive load to having hundreds of modules in a crazy tree of dependencies?

What's not to like? The fact that every one of those dependencies is an attack vector when one of those package's maintainers gets compromised / project hijacked / bought off by some black hat operator.

It's easier to keep an eye on a small number of trusted parties than 200 random strangers.

You thought this SNAFU was bad...

That has nothing to do with how granular packages are. The npm is broken as it allows such things. Look at this spectacular example of providing high security for users: https://www.npmjs.com/package/kik

Someone has to do this manually?! If the package is not popular, no one cares? What happens if I send them an email and provide the same package with the same API (not trademarked and MIT licensed) but break it a bit on every update?

No one knows.

when those packages are not under your control, it has everything to do with how granular they are and by extension how many you depend on and thus have to trust/verify.

when was the last time you rechecked the entire source of a package you depend on after updating it?

> By relying on these small rigorously tested libraries they are avoiding the need to test their own code, and thus avoiding basic null check or conversion errors.

In practice, many, if not most, of these one-line modules don't even work correctly, and it is difficult to get people to care to collaborate on fixing them instead of just replacing them with a new one-line module that works slightly better as the concept of complex collaboration to perfect one line of code is so foreign to a generation of developers who would rather just throw code to the world and never look back (an issue made all the worse as tiny bugs in these one-liners can almost seem dangerous to fix when you aren't really sure how they are being used: maybe it actually is better to not fix anything and just have people rely on the replacement better module :/).

Odd, the ones I use all have tests, thousands of downloads, and active bug trackers. If I reinvented the wheel, or copy pasted, I wouldn't get those things.

Doing a for loop is not equal to re-inventing the wheel.

Words and phrases have lost their meaning nowadays.

By the JS community's standards if I wanted code to capitalize the 3rd or sometimes 4th letters of a string, they would be 2 different npm modules.

Maybe they didn't forget how to program, as the article implies; maybe they never knew how to program in the first place.

Writing your own solution, when you could use a common widely known solution, exactly matches both writing your own left pad and inventing your own wheel.

> By the JS community's standards if I wanted code to capitalize the 3rd or sometimes 4th letters of a string, they would be 2 different npm modules.

If there was, it would have been published and people would be depending on it. Instead, people use the stdlib first, then big lego bricks like underscore, then (when they don't want the bulk of big lego bricks) smaller lego bricks. A single capitaliseMap() might be in there but it's a much more specific case than padding.

A 1-line of code module matches your needs exactly just for today. When you have 100 such modules which don't quite work and don't quite need after a month(quite a long time for a JS project to exist untouched), then your dependency hell shows its ugly head.

If you had years of experience in a wide array of technologies and languages, you'd be aware of that which is, and has been, obvious to the rest of us for the past 2-3 decades.

> A 1-line of code module matches your needs exactly just for today.

Nothing about function length determines utility. There are many one lines that match many people's needs repeatedly.

> When you have 100 such modules which don't quite work

This is worrying. How are you picking the modules you use? 100x popular modules from npm - with git repos, unit tests, READMEs, and hundreds of other users - beat 100x functions implemented in-house for NIH (or more likely, 100x functions copy pasted from Stack Overflow).

> If you had years of experience in a wide array of technologies and languages

Please don't assume things about other people. It's very rude, and it makes you look bad when you're wrong.

Worry not my friend. I'll take the risk of being wrong. Up to now, it seems that there's consensus on the fact that the JS/frontend world is the worst offender when it comes to engineering robust software.

At least there are JS programmers that seem aware of it and agree that this has to change. So there's still hope I guess.

I think there are people who are aware of the problem, but that problem is people reinventing the wheel a thousand times, and relying on copypasta and needless work rather than package management.

> JS/frontend world is the worst offender when it comes to engineering robust software

I completely agree. I still can't understand why the language that we depend on for increasingly large-scale front-end work still works like this:

[1, 10, 2, 21].sort() // returns: [1, 10, 2, 21]

It's 2016, software engineering is purported to be a profession, and we still have the worst tools imaginable.

Yep. And very large software developers have created languages that compile to javascript ... it is madness but it solves their problems.

Try

.sort(sorts.numerical)

Using npms 'sorts' module

I think the only thing you highlighted is why weakly typed languages have issues with readability. Looks like it sorted it fine to me.

But surely the really concerning part is despite their very wide usage, these dependecies turned out in some cases to be poorly coded functions indeed. Throughout this whole thread are examples where these one or two line "modules" have poor runtime performance, miss key cases in the one single thing they claim to do, and in the very worst cases, those small "modules" themselves are depending on many more "modules".

Diseases turtles all the way down.

> Throughout this whole thread are examples where these one or two line "modules" have poor runtime performance, miss key cases in the one single thing they claim to do, and in the very worst cases, those small "modules" themselves are depending on many more "modules".

Where? I've only seen the exact opposite: folk who don't believe in package management write their own naive 'reinvent the wheel functions' with those issues. Nobody I can see is quoting actual npm modules.

The OP's example ( is-positive-integer) has tests, and is only three lines long, and is on major release version 3.X, because despite all that, it had serious problems in versions 1 and 2.

"Serious problems in versions 1 and 2" suggests that it might be a good idea not to write your own version 1.

But to be explicit about the history: version one[0] had dependencies. Version two[1] had no dependencies, and a comment on github suggests that it might be wrong but I can't immediately see how. Version three[2] is a different implementation, and adds another export.

[0] https://github.com/tjmehta/is-positive-integer/commit/3003f9...

[1] https://github.com/tjmehta/is-positive-integer/commit/b47e90... - it should return false for 0, so I'm not sure what the commenter is getting at

[2] https://github.com/tjmehta/is-positive-integer/commit/3cdfe1...

Case in point: the left-pad package has quadratic complexity.

Depends on the JS implementation in question. And I think node uses one that optimizes string concatenation, possibly making that the fastest way to do it.

Yeah, as far as I know in modern JS engines += is actually faster than using an array.

[citation needed]

It's really nice how we once again depend on "I think some guys do it like this" for performance. What happened to standards?

What's next? SEO for code?

> basic null check

What's funny to me is that leftpad(null, 6, ' ') outputs " null".

Am I right that len(" null") here is 5, so it's working incorreclly?

Thought it was a typo on my part... but HN seems to be trimming running spaces.

HN isn't: your browser is. Runs of whitespace are collapsed in HTML to single spaces unless specifically styled otherwise. You can verify that they're there by viewing the page source.

I like to use non-breaking spaces for that kind of thing, although they still might collapse that? At least it doesn't wrap non-breaking spaces.

No, they wouldn't collapse that: non-breaking spaces aren't considered whitespace in HTML. The side effect of that is that if you copy some text with non-breaking spaces, you'll get non-breaking spaces, so something that looks as if it contains spaces won't necessarily behave as such. In HN, if you need to quote something where space is significant, you're best off formatting it as a preformatted block by indenting it:

    Here is    some    preformatted text.

It might break up the flow of the text, but if something like whitespace is significant, that's probably a good thing.

I think the joke here is that javascript has a long history of thinking that this:

    null

is the same as this:

    "null"

...Which it isn't.

I think it's a very valid (and concerning) point. JavaScript is made in way that one fears to implement isPositive or isArray.

> Yes they are, in the same way that a book in which every page consists of a single word is easier to understand than one with more content per page.

A more apt metaphor would be separating a book into many small paragraphs that each serves a single purpose makes the book easier to understand. Regardless, the book metaphor misses a lot of the nuance.

Of course taking the approach to an extreme would be detrimental. However, a vast majority of these small modules that are actually widely used consist of functions that may seem trivial at first sight, but actually contain a lot of corner cases and special considerations that a naive inline implementation could miss.

Sure, there are also plenty of trivial one-line modules on npm that don't fit into a such a description, but those are simply side effects of the unprecedented popularity of the platform and its completely open nature, and shouldn't be used to infer any kind of general trend towards "hypermodularisation" because very few reasonable developers would ever import them into their projects.

> A more apt metaphor would be separating a book into many small paragraphs that each serves a single purpose makes the book easier to understand

No, that would not be an apt metaphor for the problem he is describing.

> Regardless, the book metaphor misses a lot of the nuance.

Not if you are trying to understand the point he is trying to make.

Is there any cost to creating modules, uploading them to npm and using them in other projects ? Clearly there is, as is shown by the world-wide breakage from yesterday. This is probably only one of such failure modes.

The point is, breaking everything into micro-modules can have benefits and costs. Ultimately, it is an engineering trade-off.

Now, it is possible that npm modules are at the right level of abstraction, even though other module systems in the world are not this granular. If this is due to the special nature of JavaScript then the point must be argued from that perspective.

> No, that would not be an apt metaphor for the problem he is describing.

That's because I don't agree that the problem he is describing exists, or at least not to the degree he's describing, which was full of hyperbole.

> Is there any cost to creating modules, uploading them to npm and using them in other projects ? Clearly there is, as is shown by the world-wide breakage from yesterday. This is probably only one of such failure modes.

This was a failure on npm's side by including a functionality that allows users to trivially remove packages from a package management system that is used by hundreds of other packages, something that most major package management systems have decided was a bad idea.

> The point is, breaking everything into micro-modules can have benefits and costs. Ultimately, it is an engineering trade-off.

Agreed. And I don't think nearly as many people are erring on the extreme side of this tradeoff to the degree that he's describing.

> This was a failure on npm's side by including a functionality that allows users to trivially remove packages from a package management system that is used by hundreds of other packages, something that most major package management systems have decided was a bad idea.

That is emphatically not the problem. The author of those modules could have just as easily modified the code instead of deleting it:

    function leftpad(str, len, ch) {
        // NPM won't let me delete modules, so here you go
        return "";
    }

Now you'd have an even harder time figuring out what happened to your code that you did if the module just disappeared. What you're asking for is basically for the repository to adopt a policy of "all maintainers must be given free rein to get their modules correct and keep them maintained, but they shouldn't have the ability to do anything that causes problems downstream" which is impossible and frankly silly.

The problem is the dependency, not the precise mechanism by which that dependency screws you when it goes bad.

You can't republish with the same version number, so you would be protected if you're pinned to a specific release.

At which point you don't get bug fixes. This is an old problem and there is no good solution.

You get bug fixes when you upgrade, deliberately.

This is what everyone should do ideally, but that means your solution is conscientiousness, and good luck spreading that across all your dependencies.

> for the problem he is describing.

Because it is a strawman.

> By focusing on the small-scale complexity to such an extreme, you've managed to make the whole system much harder to understand

Well put. I've noticed a curious blind spot in how people account for complexity: we count the code that 'does' things much more than the code that glues those things together. This distorts our thinking about system complexity. A cost/benefit analysis that doesn't consider all the costs isn't worth much.

An example is when people factor complex code into many small functions, then say it's much simpler because smaller functions are easier to understand. In fact this may or may not be true, and often isn't. To get a good answer you must consider the complexity you've added—in this case, that of the new function declarations and that of the calls to them—not just the complexity you've removed. But it's hard to consider what you don't see. Why is it so easy not to see things like this? I think it's our ideas about programming, especially our unquestioned assumptions.

The complexity of glue code goes from being overlooked to positively invisible when it gets moved into things like configuration files. Those are no longer seen as part of the system at all. But of course they should be.

I just saw this insanity a couple of days ago:

https://youtu.be/l1Efy4RB_kw?t=343

He refactored a Java application into 1400 classes!

And to add to the mess, until npm introduced deduplication every dependency recursively included it's own dependencies as sub folders, so you get a bajillion tiny dependencies over and over (and sometimes with different versions to boot).

Frankly some of the comments I'm seeing really do reinforce his point: people really don't know how to program. I suspect this is it the flipside of Javascript being popular and accessible - people can churn out products without really knowing what they are doing...

Sorry for the nitpick, but "hyperabstraction"?

left-pad isn't even slightly abstract.

It's not.

The hyperabstraction is in how even tiny functions like this, isPositive, isArray, etc. are being abstracted and turned into individual modules.

Again, maybe I'm nitpicking, but turning something into modules doesn't seem like abstraction... you could call it encapsulation, or modularisation, or maybe extraction.

But abstraction? I don't see how that word is connected to what's happening. Maybe you can explain it.

Reply on news.ycombinator.com