Hacker News Reader

Comment by com2kid | Hacker News Reader

Since 2012 we have, as a field, learned better.

You see, we all realized that just building code wasn't enough! What your 100 line JavaScript microservice really needs to be accompanied by is a build file that determines how to create an entire Linux VM. Also, because scale, your project also needs to know how to scale that VM out across multiple clusters!

The only answer of course is more build systems!

Oh but JavaScript sucks, so make that Typescript, so shove a transpiler in front of your scripting language. Because somehow we've gone down the path of needing build systems for interpreted languages.

I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/

(I legit like Babel and TS, but you have to admit it is /weird/ that we're going through the complexity of having a build system but are reaping none of the performance benefits!)

View on HN

From my perspective, the main reason for most browser JS projects having a build system is to essentially concatenate modules and to transpile newer JS features so they work on older browsers. That’s really just not as crazy as you make it sound. I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy.

> I don’t know about the Linux VM stuff being part of a browser JS project, maybe I’m misunderstanding that part but it does sound crazy.

A bunch of stuff I see comes with a dockerfile these days.

While I get building in docker for reproducibility, kind of -- shipping with one is a pretty big red flag. It means that it's likely to be very painful to port the code out of docker and integrate it into some other build. Which means that it'll be painful to integrate it with other code. Leave the build environment set up to the users. Embrace build diversity, it makes your code less fragile.

At my job, a colleague shared a solution for tunneling into servers that involved having a Docker image running for each server, with some tunneling tricks inside the image.

Those are the times I wonder whether I’m getting old. I kept using a shell alias that does ssh -ND, even though that requires maintaining a config file.

But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment?

> But on the other hand, is a dependency on having Docker available really worse for portability than having a dependency on having a reasonably standard Unix environment?

I often need to run it somewhere other than docker -- possibly integrating it into a different program.

Docker doesn't make this impossible, but it's a sign that portability was not considered when the code was written, and I'm in for pain.

> It means that it's likely to be very painful to port the code out of docker and integrate it into some other build.

No it wouldn't. Dockerfiles are stupid simple. A bunch of ADD/COPY commands to add files, and a bunch of RUN commands that are basically wrapping shell scripts.

Plus, find me a modern build system (CI/CD system, rather) that doesn't natively support Docker. It's become a lingua franca at this point.

I just wish it was faster on its own without me having to do all the legwork of maintaining lower layer containers etc. to avoid lengthy package install times.

I had a problem yesterday where I had so many docket images laying around from builds that starting a new container pegged my CPU at 100% and slowed everything in a docker container to a crawl.

A colleague predicted the problem and I learned about one more bit of random fragility that exists in today's ecosystem.

> I had a problem yesterday where I had so many docket images laying around from builds

$ Docker system prune

It makes as much sense to complain about Docker because you let images accumulate in your system as it would make to criticize file systems because you never emptied your system's trash bin.

My file system doesn't peg my CPU when it gets 10% full.

Why the hell should the very existance of unused docker images cause the single image I am using to peg multiple cores?

And no warning was given. Heck if my hard drive gets close enough to full that it's gonna be a problem my OS tells me.

> No it wouldn't. Dockerfiles are stupid simple. A bunch of ADD/COPY commands to add files, and a bunch of RUN commands that are basically wrapping shell scripts.

Yes. And the code sitting beside them starts to make assumptions -- so when I try to port it to Android so that I can run it under the NDK, for example, it turns into a shitshow because I discover that the code depends on the quirks of the docker environment, and I need to untangle a whole bunch of issues to just get it to build on anywhere.

Containers -- dockers, jails, zones, etc -- are useful tools, but I'd rather see them left up to the people doing deployment, rather than the people shipping libraries. As a library author, I don't know where people want to run my code, so I serve them better by keeping the code portable.

I was thinking server side.

Client side Babel makes sense. And packagers make sense. And it all unfortunately makes sense, but that doesn't make it good.

(Ok being able to try out new syntax with Babel is cool)

What would make it good? If there’s a real problem, and you have a real solution that makes sense, what’s not good?

It's the classic question of "Reform or Revolution?"

These systems were built as incremental mitigations to old problems. I don't think it'll change until they collapse under their own weight and people switch over to something else completely.

Technically containers are not VMs

And re: build systems for interpreted languages, what would the alternative be?

Asking Firefox and Chrome to integrate a language they didn't design?

Making it a runtime module like MoonScript, and letting everyone who loads the page pay for compilation, on battery-constrained computers, millions of times over?

>Technically containers are not VMs

Some build systems do build proper VM images, like AMIs.

You're talking about a cloud service provider, whose bread and butter is selling you time on their VMs. Of course they manage machine images.

Your remark is funny, as I was looking at Bazel for TS last night. It seems that Bazel and PNPM don't work together, so I pretty much just have to punt on it, unfortunately (yes, I have a monorepo).

Alas, it seems like a ton of extra effort to have BUILD files everywhere that repeat the imports of their sibling TS files. And on top of that, how am I going to ensure they stay up to date? What if someone forgets to manually add a dependency to the BUILD file after they've already done an `import {} from ...`? What if I don't want to vendor dependencies?

Unfortunately, it seems like Bazel + TS could be really cool, but the amount of extra effort won't pay off because there are foot-guns all over the place.

> I use all the above tech and get why it is there, but... seriously as a field this is the best we can do? :/

I think it can be done better, and above all, much simpler.

I think "redo" looks like a very simple, good solution - here is the documentation: https://redo.readthedocs.io/en/latest/

Of course it is only a tool and it does not convert an overcomplicated bloated system into a three-liner. But unnécessary complexity in build systems sucks big time, and interferes with decent testing, so I think this is a real gem.

> (Disclaimer: my current implementation is not as fast as make for some things, because it's written in python. Eventually I'll rewrite it an C and it'll be very, very fast.)

Lol. Whatever, python community...

Anyway, I don't think redo really provides an answer to the insanity that is the js build system(s). I'm also a bit of a DJB fan, like python as a language, etc. However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system. He uses SQLite instead of the file system as a database paradigm favored by DJB.

DJB's software is all about technical correctness and simplicity of design. There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it.

The whole "I'm moving my python code to C tomorrow" thing is becoming a bit of a smell IMHO.

Shell script or a compiled language would be preferable in my opinion.

> However, this implementation in python... eh... I'm not sure about it. It's annoying to require someone to have some particular version of python to utilize a simple build system.

I think that Python as a draft/concept language, for getting the semantics right, and after that rewriting it into a standalone C program is sound, and that is the stated plan for the project.

I think what originally matters is not how it is implemented, but but how it is defined, what are the exact semantics, this is what was the Python development about. Because it is very simple and around 3400 lines of python, creating additional implementations should be not too much work, and my understanding is that a C implementation is the next planned step.

> He uses SQLite instead of the file system as a database paradigm favored by DJB.

Speed is one issue here, it also needs to support efficient multi-processing. Based on my own limited experience with something similar, my personal favorite for this kind of thing is LMDB (https://en.wikipedia.org/wiki/LMDB), which is well-tested, but I guess it would be very hard to make that into a portable version which runs, for example, on Windows, too.

> There's nothing wrong with this per se, but it wouldn't surprise me in the least to run this on some version of python that it isn't quite coded for and have it fail in some weird way without me realizing it.

From following the list (https://groups.google.com/forum/#!forum/redo-list) and blog, my impression is that the main author is a bit of a quality and reliability fanatic. So, for that kind of project, exactly the right person. He provided a test suite which can be used to test alternative implementations, as there are already a few around.

> Shell script or a compiled language would be preferable in my opinion.

There exists a basic (non-parallel) implementation in shell. shell is however going to be a bit too slow for a "production" implementation for large projects where speed matters. My favorite language for this might be guile (I think it matches Guix well), or perhaps Common Lisp. For maximum portability, I believe C is really the best thing.

I'd have nothing to say against an ultra-fast implementation for POSIX systems, written in Rust which calls into LMDB. But while this might be beautiful from a technical perspective, I don't think that there will be a noticeable speed difference in normal use.

It's more like the very least we could collectively achieve while remaining functional.

Reply on news.ycombinator.com