I got the J language working on OpenBSD | Hacker News Reader

I got the J language working on OpenBSD

briancallahan.net

148 pointsingve41 comments

Thank you! Now OpenBSD routers will have a powerful array language available for statistical analysis of network traffic. :)

I think there is a J book on statically analyzing network traffic

Link to the book in question https://www.springer.com/gp/book/9781846288227

http://libgen.rs/book/index.php?md5=4FA72167BDF2DC436A22F5C0...

One thing that I noticed that we tend to not like in OpenBSD ports is that -Werror was in every compiler invocation line. That had to go.

Googling “openbsd werror” doesn’t show anything, why is this?

Werror is great for development but not actionable if you're just trying to install it as a user.

This is true, but the whole things sounds like anathema to the OpenBSD ethos, as I know it, which seems obsessed with a sort of functional correctness and consistency.

Enabling Werror sounds like exactly the sort of thing that'd be encouraged by default, and only disabled on ports that couldn't be patched to fix it. Or at least, enabled on ports that can have it enabled.

(that said, I know that the vast majority of software won't compile with Werror, so, ?)

I'm not involved in OpenBSD but I don't see the value in preventing an end user from installing arbitrary packages just because the compiler was updated recently. The person being notified of the error is the wrong person.

True, but OpenBSD seems to some extent value correctness over "usability". I can imagine the argument being made that if something changed and caused the package to stop building, it should be fixed, because the warning was potentially significant.

Since they do want the OS to be practical and usable, though, your argument also holds true, and seems to take precedence here. Just kind of surprised they'd make a policy of explicitly patching out upstream's Werror, since it implies it's "supposed" to work with no errors.

OpenBSD's ports tree gets compiled using multiple compilers (clang/gcc) and compiler versions (base 4.2.1, ports gcc8/11), across many different architectures. In most cases when an upstream has added -Werror to a Makefile, they have not considered this. It's good for testing, but not for reliably compiling software.

You are speaking way too conceptually and not looking at the actual warnings that compilers report.

Some compilers will warn about:

   while (true) { ... }

having a constant condition. The fix is to write:

    for (;;) { ... }

to nobody's benefit.

I've used languages that warn if you use snake_case instead of camelCase, because it violates their preferred naming convention. Or if you accidentally use a tab somewhere, it'll warn about a mix of tabs and spaces.

None of these matter for correctness, yet you're advocating to break the build for correctness.

> The fix is to write: > for (;;) { ... } > to nobody's benefit.

Idiomatic constructions benefit whoever would read the code (along with choosing and following to a style guide).

-Werror can be useful for CI, but elsewhere it is obnoxious. Warnings should be warnings, errors should be errors. If you always turn warnings into errors then you remove a useful tool.

Warnings do not mean the program is incorrect, and lack of warnings do not mean the program is correct. It is just heuristics, and changes between major versions and different implementations, even while the program execution behavior is exactly the same. OpenBSD cares about portability. A well implemented program should work correctly with a good range of different reasonable compilers, and -Werror really throws a wrench in that.

Imagine you want to update the compiler to a new major version that has some corner-case correctness improvement or some security mitigation feature - but it also has a new warning. Can't do that! Unless you change the source of a bunch of libraries and applications which you didn't write. Mistakes have been made fixing warnings, like the infamous Debian openssl fix that removed most of the entropy from generated ssh keys.

If you care about correctness, you pay attention to and fix warnings where appropriate, without forcing yourself by making all your builds on different compiler versions/implementations just fail.

There was a time that CPython wanted to add a deprecation warning, but some popular libraries ran tests with the equivalent of -Werror during a regular build/install (trying to be super-correct best-practices etc), so adding a warning would break compatibility with most of the ecosystem, so CPython couldn't just add the warning using the long-established system. Instead they had to come up with a new system for warnings shown in some circumstances and not others, to effectively trick these super-best-practices projects into still working.

There is nothing wrong with -Werror, if you control the exact set of diagnostics!

The problem are the "surprise me!" warning options (-Wall, -Wextra) of GCC which bring in diagnostics which differ from version to version. So then with -Werror, the language has become a moving target: what is considered a bad construct that stops the translation is changing.

The problem with not using -Wall and -Werror and just specifying the exact diagnostics you want, together with -Werror, is that you don't benefit from the discovery of defects brought about by new diagnostic options.

The fix for that is to have an ongoing thread of activity in your project dedicated to exploring new compiler options (supported by a special build configuration).

gcc has been shoving warnings into -Wall or -Wextra that actually belong to a linter for years, for example -Wmisleading-indentation.

Bug free software that compiles warning free has to be updated because of the stylistic preferences of some RedHat person practically with every new gcc release.

For KPI harvesting developers of course "fixing" non-issues is financially attractive.

If a user tries to compile with a compiler that gives a new warning, what do you expect the user to do about that?

I know nothing about OpenBSD, but one reason to avoid -Werror in shipping code that I've heard is that the set of warnings is a moving target.

It's easy to imagine having a compiler upgrade then add new warnings, that cause unchanged code in a project with -Werror to no longer build, thus breaking it.

Of course it can be argued that such "exposure" flushes out bad code with a vengeance, but it can also be considered to be annoying.

The new warnings also tend to be increasingly asinine, and I remember an instance where two warnings happened to be in direct contradiction to each other, making it impossible to satisfy them both without some extensive and absolutely useless rewriting of the code.

I think one was about superfluous parentheses, and the other about not parenthesising an expression because it assumed the programmer would be too stupid to know the precedence of operators.

Rubocop is so bad about this. I can't believe people still use it. I spent 2 hours refactoring perfectly legal code to satisfy a code review and then found the suppress warnings invocation and ended up shipping that.

As someone in a DevOps role who has three times been the steward of major migrations between baseline OS versions, I have often been frustrated at teams who have hard-coded -Werror into their build configs, thus forcing me to jump through hoops to patching it out because the new compiler has some trivial complaint about this or that and the build is uncompileable on the new target.

I understand the noble goal of being warning-clean, but I wish there was a way to implement it in such a way that it could also be globally disabled. Then I could get a top level build, but the local resolution of the issues would still be on the owners of the code.

Zero warnings is the kind of thing that should be enforced as a pre merge check rather than a build showstopper.

Yeah, 100% agree. Or even better, for incremental improvement, have it enforce no new warnings.

But all of this requires adding complexity and branching to the system. It can be done and is worth doing, you just have to choose your battles.

In the case of the team moving to a new compiler, what are you suggesting? We can’t merge in use of a newer compiler until there are zero warnings in all our products? Who “fixes” all those code bases? Who touches the pieces of code that were last touched 5 or 10 years ago?

Various solutions are possible, including the obvious one of just removing -Werror. But the point is that this decision should be made by the team that owns/maintains the code, not by some centralized devops group.

> It's easy to imagine having a compiler upgrade [...]

Not in OpenBSD it isn't.

Since the move to Clang/LLVM the compiler in OpenBSD has been pretty up to date.

The user might otherwise compile it on a newer compiler with new warnings. You don't want such new warnings to break the build for users installing the software.

It would be interesting to see a writeup comparing J and the open-source implementation(s) of K. Which would be better for a complete novice at array programming?

J is definitely more batteries-included than the open source K implementations.

There's also a wealth of documentation and books on J: J for C Programmers: https://www.jsoftware.com/help/jforc/contents.htm

Learning J: https://www.jsoftware.com/help/learning/contents.htm

And the New Vocabulary "Nuvoc" on the wiki: https://code.jsoftware.com/wiki/NuVoc

Edit: spelling.

J is also way harder to learn in my opinion due to it having many times more operators. K2 and Q also have official books.

I'd recommend APL2 and the GNU APL implementation instead. The reasoning is you can follow through the excellent Ken Iverson Turing Award paper [0], and, while using non-ascii is a small upfront cost, it IMO makes array languages much more readable.

[0]Notation as a tool of thought, https://doi.org/10.1145/1283920.1283935

I would recommend Dyalog APL over GNU APL, it is much more modern.

That's proprietary. And modern isn't an advantage in itself, unlike open-source for someone requesting open-source.

I know it's proprietary, it isn't my language of choice, but it's a lot better than GNU APL.

If they really want something open source and good, J, open source ks, or BQN will be better. But if you're suggesting an APL dialect, Dyalog is much better than GNU despite being proprietary.

K is the easiest popular array language due to its array model and its commitment to simplicity. Gets you into the habit of writing out strings of symbols instead of names, and although it holds your hand less, I'd say it's great as a beginner language. With the existence of new docs like the oK docs(https://github.com/JohnEarnest/ok/blob/gh-pages/docs/) and the wiki(https://k.miraheze.org/), it is getting much easier to learn K.

It feels to me like this is a bit of an indictment of the J build system. I wonder how hard it would be to port it all to something more standard?

Thirty year old codebase with few developers, built on multiple platforms. It's about what you'd expect. But I believe there have been some improvements made in the past five years or so (the source was released under GPLv3 about ten years ago).

BEST, llc has built it with Tup (see https://github.com/iocane/unbox), so changing the build system is at least possible.

Edit: there's also a long-open PR to move to CMake at https://github.com/jsoftware/jsource/pull/20.

The changes made were pretty minor. It looks like one #ifdef __linux__ needs to be looked at, and some better detect/#ifdefs around AVX and SSE3. At least to make it work on unix-y systems.

Indeed, there is a whole lot of room for improvement. In fact, a makeover is in the works. There's not a hard timeline, but it will take a bit of time to weather anything new before it fully replaces the current, somewhat cranky, one.

Long time ago, Kx used to provide binaries for FreeBSD. I wish they would provide binaries for NetBSD (and OpenBSD) today. They have always had binaries for Solaris. Then they started providing binaries for RPi. How difficult can it be to build binaries on BSD, BSD is so easy to install; perhaps it could potentially expand community interest. Meanwhile we use Linux compat, but thats been removed from OpenBSD.

Thank you! this is what senior programmers are for, you-all

Reply on news.ycombinator.com