Who killed the network switch? A Hubris Bug Story
cliffle.com
cliffle.com
I recommend leafing through it: https://github.com/oxidecomputer/hubris/blob/b44e677fb39cde8...
Disk space for source code hasn't really been a problem for forty years (binaries definitely), and yet we are still being stingy with variable names.
Each and every time there was just insane excuses that boiled down to only a few people wanted to write C++ and they found infinite ways to justify not doing things, like all us humans do.
My favorite was the "don't write comments because as soon as you do they're out of date."
Even though this is recognizably trollish tripe at the extreme it was implemented, I still find it hard to ignore that brainworm in my day to day coding.
It’s the same deal with compilers and computer hardware. They work so well that they become a blind spot, and yet I’ve had both fail me.
A better version of that is “if the code and comments disagree, there is a good chance that both are wrong”.
If code and comments get out of sync, at best someone has been careless, and you need to be on the lookout for more carelessness in changes made at the time. At worst changes have been made without fully understanding what is going on, so there is a huge can of edge-case worms about to burst open.
A lot of commercial code is like this, regardless of the language; there just happen to be people trying to sell new languages. They desperately want to convince you that this is all the fault of C and it's nebulous "ethos," and if you just use their new language, it will all magically go away and possibly improve human rights somehow.
And don't even get me started on trying to text to speak your `mCtx` or whatever
I'm not saying TTS is there (I have no idea) but from an ideal point of view, to have it be equivalent, surely it would read 'm context', which then isn't so bad (maybe m is clear from the ..err.. context).
Although more expensive, it worked wonders for me with text containing units of measurement (°F, °C, m/s, etc).
IDK, I've yet to see a good argument for not having your variables being generally pronouncable (at least, in a way more or less consistent with the native language of your code base or English). I'm just going to lazily parrot the old adage that "code gets read more often than written, so prefer making it easier to read than write"
That said maybe `mCtx` wasn't the best example, but in general anything other than fully pronounceable words in a normalized format (PascalCase, camelCase, snake_case, kebab-case, etc) is a huge pita for language models in our experience. Things like `i` or other single letters are generally okay.
But imagine things like "wait did you mean the letter 'm' or the unit 'em'"
In short, technology cannot fix people problems.
To be fair to C, it was designed long before we as a society really appreciated that. So there is a lot of old code whose authors just couldn’t know better. And some (probably few) will even have managed to write readable code! But that doesn’t mean that languages can’t encourage good behavior and discourage bad. (I’m not even sure where I come down wrt Rust here) And that will (somewhat reliably) end in better or worse code. Not necessarily in a reliable way. But it’s e.g. really hard to write python code where local control flow isn’t reasonably obvious (since scoping is enforced by white space). Not impossible of course
atoi()? strrchr()? srand()?
I’d argue the obtuseness of the standard library function names at least influence the legibility of the programs written against them.
I suppose you are too young to remember that back in the day there was no IDE helping you with autocompletion, thus one character less in the name is one character less to type.
Furthermore, I'm ready to bet you are from the USA, and you forgot that most of the world (even the programming world) does not speak English natively, thus "SeedRandom" is NOT clearer than "srand": it's just as cumbersome and longer to type while reading character by character on a paper manual.
Well then it’s good that OP didn’t claim that C the language causes people to write such code. They said “The C ethos”, not “The C language.” It’s not about the language’s technical requirements, it’s about what’s idiomatic in a language, how it’s taught, and what style is used by the vast majority of the existing corpus of code written in that language.
Look at the C standard library’s function names, vsnfprintf/strdupa/acosh/ftok… Compare it to something like Objective-C at the other extreme, where method and variable names tend to always have fully spelled out identifiers with no abbreviations and a full description of what’s being done (`- [NSString stringByAppendingString]`, etc.)
Is it due to some technical requirement? Is stringByAppendingString illegal in C because it’s too long? Is strdup illegal in ObjC because it’s too short? Of course not! But why do we see this everywhere so consistently? Why does C have short indecipherable function names and ObjC have such long ones, if the language doesn’t require it?
Because idioms matter. If you’re learning C, you’re learning the way it is typically taught. You’re reading other C code. You’re encouraged to program the way other C programmers program. You’re likely using the standard library a lot. Likewise ObjC.
This means two things:
- Yes, in a very obvious sense, it’s not the language’s fault, it’s the programmers’s fault.
- But also, paradoxically, it is the language’s fault, because a language is not just a set of syntax in a vacuum, it’s also a corpus of existing code, a set of idioms, a community of people, and a way of thinking of things. C absolutely causes people to write hard-to-understand code when viewed through this lens.
Your comment has been written in so many ways in so many threads discussing programming languages, it’s absolutely tiring. Yes, you can write terrible code in any language, and you can write great code in any language (well almost any… probably not brainfuck.) Nobody is arguing that. When we discuss whether one language is more readable than the other, we’re always talking about the qualities of typical code you actually see in a language, and about what style of code that language encourages.
Same reason why Pascal has symbols limited to 10 characters and doesn't preserve case - because original implementation mapped identifiers into 10 6-bit characters packed into 60 bit word.
Similarly some of the oldest Common Lisp functions retain aspects of encoding characters to fit into 36 bit words efficiently.
It should not be the first language freshmen are taught in universities. All the effort will be wasted on teaching basic programming instead of the actual language and good programming habits.
Pointers and the memory model are core concepts and should be taught early on. The worst programming books I have seen introduce them in the very last chapter, often called "advanced language features" or something similar.
It is not a compiled scripting language for Unix systems either and should not be programmed with a text editor on a remote machine. The programmer should have a proper development environment installed locally.
Valgrind and and Clang's sanitizers should be used extensively. There is no such thing as partial credit when it comes to memory correctness.
If a certain QuakeNet IRCop is reading this, thank you. Your uni C course managed to avoid all the pitfalls above.
"should not", "They should have" ... well, it had to! And they didn't have.
There is no point in trying to forget where C comes from.
The example here is very much C-style (short, lowercase, underscore). The list of argument is called "args" (short and descriptive). It is not called argumentListArray.
It is not a coincidence that math have very short and well known identifiers. The rate of change in horizontal position is called x'. It is not called posHorizontalChangeRate simply because the latter is harder to read.
There is, of course, such a thing as having too short identifiers.
You’re proving the original point - it should be named “arguments” because that is what it is.
Saving … 5 bytes? by naming it “args” instead is exactly OP’s point.
It is more about if your code needs to parse an internationaly formated phone number from a string into a struct. (or pick anything else specific only to your problem domain) How do you call that function? Do you call it prsphi? (prs for parse, ph for phone number and i for international) Or do you call it parse_international_phone_number? Or maybe international_phone_string_to_phonenumber_struct? (or something similar in CamelCase)
That is where the difference starts to matter. And I for one don't want to read code with many prsphi's around.
I have never worked with phone numbers outside of a class assignment so prsphi is not something I would understand. However if I was in a codebase that worked with phone numbers often I'd probably get to know it and like the shortness.
That's a false dichotomy; you could call it for instance parse_int_phonenum, abbreviating "international" and "number" to the still understandable "int" and "num" (and smashing together "phone number" into "phonenum" since it's a single concept in this code base). It's nearly half the length of your suggestion, while still being understandable at a glance.
In any case, I can remember when I started programming and no I didn’t have any problem remembering that arg is short for argument. I’ve seen acronyms and shortening of words all my life before programming after all. However I do remember being confused by the concepts argument or keyword argument in the abstract.
Short said, I really think this is making a mountain of a molehill. If someone doesn’t know what it means the answer is “arg is short for argument” and then you move on.
Who do they ask? They have just interrupted someone else's flow to ask - worse they might ask the wrong person and interrupt more than one persons flow in getting an answer. There is a high cost to even trivial questions and the more of them someone needs to ask the worse.
The point is there is a balance you need to find that balance.
I take pedagogy very seriously and work very hard for both experienced and inexperienced people to understand what I do, but this is just ridiculous. This is a non-issue.
I can understand it from way back when. People were moving from assembly and other languages where everything was terse by requirement so variable/function/other names were short out of habit, though even then your assembly should have had comments about what was in each register & why etc. Also, comments were relatively terse because people were working on small screens and in extreme cases were concerned about the size of code files. None of this is adequate excuse for doing things wrong now, decades later, but such habits have momentum: people with the habits write books and local style guides, and also short names are baked into the standard library, so new people are “infected” by the habits, and so it rolls forward.
The standard screen size was 24 lines of 80 columns, and you couldn't expect a reader of your code to have a terminal with higher resolution than that. So the standard is to make your code fit in 80 columns; using longer identifiers means statements will end up taking multiple lines, which means less code can fit on the screen, making it less readable.
Even today, terminal emulators like Gnome Terminal still open by default with 24 lines of 80 columns, so there's still value in making code readable in that resolution (though nowadays, the main advantage is making it easier to read code side-by-side in multiple windows).
"My computer is a LITERAL TELETYPE, so I will enshrine typographical parsimony in every part of this OS/Language I am creating" -K&R
So is world peace, while we're saying things that contain the word "merely". :)
Nit:
// Order the task's regions in ascending address order.
//
// THIS IS IMPORTANT. The kernel exploits this property to do cheaper
// access tests.
regions.sort_by_key(|i| region_table.get_index(*i).unwrap().1.base);
[1] https://github.com/oxidecomputer/hubris/commit/b44e677fb39cd...
Probably the best thing to do is to make a constructor method for the TaskDesc that sorts the regions to enforce the invariants. The code is evidently getting more complex over time, so packing the complexity up into methods is probably now worth spending a little time on.
This is truly a fantastic post-mortem: even an application-level developer like myself could follow it. Though I am in the middle of Rust in Action at the moment so I was primed for this type of content (:
And of course it’s always a pleasure seeing someone else who comments their code at such a high LoC ratio. Literate programming works!
“Most of our team are based outside of the Bay Area. We do ask that your workday overlaps with Pacific Time for at least four hours.”
It's true that means that this won't be a fit for a lot of people, but that's always the case with every job.
(This is also why our job descriptions tend to not say things like "must have X years of Rust experience" and instead say things like "you will thrive in this role if you appreciate the hard-won thrill of debugging a knotty problem to root cause," because that's focusing on the direct need we have for the position.)
We do ask for some overlap in working hours with the US, which does make it harder the further away you get, but that is the constraint, not “US only.”
I'll admit that to this day Github has been a bit of sore memory spot with how they publicly were all "we hire internationally and remote" but for a long time wouldn't publish that "internationally" meant "here's a small list of countries that applies to".
Now if only I believed I had a chance to pass the interviews :D
The other project I'd be interested in working on (likely as an open source, separate project) would be bringing the dx/ux and tooling to home lab level gear.
Is there a Canadian subsidiary or is it a contract employee kind of relationship (if you can comment!)?
Our console is written in TypeScript on the front end and Rust in the back end, as far as I know: I'm not aware of Go being a meaningful part of our stack (other than in dependencies, like CockroachDB).
> Is there a Canadian subsidiary or is it a contract employee kind of relationship (if you can comment!)?
I don't personally know the answer to this, being from the US :)
1. Employees (eg people who get a 1099(?)) would seem to require a green card or visa. 2. A contract employee is more of a business relationship but avoids the above requirements (if any, I am no expert which is why I asked :) 3. A Canadian entity would issue a T4 slip and also avoids any visa requirement and has none of the downsides of doing things as a contractor. But it requires the parent to have created and registered a company here.
I currently work for a US parent with the 3rd option (we were acquired). Engine Yard (not me, but I knew people there) used option #2.
"Our own web console, CLI, and SDK consumers (e.g., oxide.rs, oxide.go)"
This caught my attention. I'd love to hear more about the motivations for crafting such a culture as well as some particular implementation details. I'm curious if there are drawbacks to fostering "openness, curiosity, and communication" within an organization? Obviously, some go for more rigid hierarchical systems, and it occurs to me that an org chart can (and probably should) be strategically decided upon, but I am somewhat clueless as to the tradeoffs. HN have any insight here?
I've experienced this in some companies I've worked for (one was a very large company where there was an explicit power structure of sorts, but it was not particularly stuck to in practice. It was a consultancy which worked on a variety of different projects, and the way you got onto projects was basically by making friends with the people selling and managing them, not so much through any particular official channel. This worked great if you were good at forming the social network needed to do this, it didn't work so well if you weren't). For other examples, there's "The Tyranny of Structurelessness" which is a talk from a feminist who noticed the same kind of thing happen in the organisations she was in, which generally rejected hierarchy as patriarchal, and you can see similar discussions around how this does/doesn't work in Valve, which also doesn't have a particularly well-defined internal structure. Open-source projects can suffer from the same thing (I would characterise some of the rust drama as stemming from a similar problem).
That said, an explicit power structure doesn't need to be hierarchical, even though that's the 'traditional' business organisation. It's possible Oxide's structure is explicit but non-hierarchical. Also, in general such an approach tends to work better at smaller scales than larger ones. Said consultancy is probably the largest company I'm aware of which still mostly operates in a freeform way there, and they do have a kind of scaffolding which keeps it kinda held in place. And of course, this is more of a spectrum than a dichotomy: even the most rigid of power structures on paper still has some implicit, more complex implicit structure underneath: it's the nature of groups of people.
Don't take this as me believing explicit > implicit in general. This is just me talking about some of the observed disadvantages of less explicit organisation. More explicit power structures have their own problems as well (and there's the related 'seeing like a state'/'legibility' issues there).
With respect to other environments that encourage autonomy, that last element -- that compensation is uniform -- tends to set Oxide apart: if an environment encourages both autonomy and stack-based ranking and compensation, it absolutely will create shadow structure. By removing the organizational tumor of variable compensation, Oxide removes the shadow structure that it creates.
[0] https://rfd.shared.oxide.computer/rfd/0003
[1] https://oxide-and-friends.transistor.fm/episodes/hiring-proc...
[2] https://oxide.computer/blog/compensation-as-a-reflection-of-...
This sounds like a really profound point. Rather than just saying "it's human nature to form shadow hierarchies", ask why they do it, to what end. If increasing their compensation is removed as a potential motivation, that must change the way people behave.
This is how some older chips used to work. Hardware page table walkers can be viewed as just a hardware implementation of such code.
You can keep soft real time perf with this scheme, just like you can keep soft realtime perf on any system with a soft fill TLB.
I say all of this as I used to be the lead for a real time kernel that would allow more than 8 regions on a task on a Cortex-M0/M3.
It's pretty easy to hit that amount with a combo of fine grain access (you probably want your RO section different than your TEXT section, you probably want your stack discontiguous with any mapped section, you've got all of your MMIO regions your task cares about which necessitate other MPU regions because the difference between normal memory and device memory is configured by the MPU), combined with breaking some apart to get greater granularity/packing like they talk about here.
Tailscale's biggest proposition is that their VPN software is, arguably, the easiest I've ever managed. There systems that are better integrated in some environments, systems that provide various niche features, etc.
But Tailscale is simplest to use.
Who in their right mind writes a new operating system today? Answer: someone trying to tackle a problem every operating system is ignoring (controllers on MB and expansion cards that the OS doesn’t and cannot control).
But yes, the idea is to gently poke fun at "wait you're writing your own OS?", which is admittedly a bit hubristic. Humor is a good way to stay grounded.
Another difference is that Hubris does not use async Rust, and embassy does. That may be something you desire either way. We don't believe async Rust is bad (heck it's used further up the stack quite a bit) but it wasn't a fit for this project.
The documentation explains the design goals: https://hubris.oxide.computer/reference/
Doesn't mean embassy isn't good too, it's just different.
As much as I understand why you'd want async on an MCU for efficiency reasons, I'm very glad there is now a good option without it.
I wonder what the Hubris team thinks of rtic and its design.
I’m sure that other folks would say the same thing here though: also good, just different.
The motivation, though, was that we wanted to write tests for the function that would run on a developer's machine using `cargo test`, and the Hubris kernel currently only compiles for the targets that we actually run Hubris on (various Cortex-M targets). So, if we want to write tests for that function, we would either need to move it to a crate that doesn't contain Cortex-M-only code, or litter the whole kernel with `#[cfg(not(test))]` attributes so that most of it doesn't compile when building tests. This felt much less unpleasant than conditional compilation. We're hoping that, eventually, other complex-but-not-architecture-specific kernel code will end up in the `kerncore` crate as well so that we can write tests for it, too, so eventually it won't be a crate with only one function...
I do think that there's room for tooling improvement to make writing host-platform tests for `#![no_std]` Rust that gets cross-compiled to another platform, but I don't have any particularly concrete ideas for what that would look like. For now, at least, putting it in a separate crate lets us have our tests --- and those tests let us ensure that some of the function's edge cases are handled correctly, so I do think it was worth it.
What could that interesting thing be? An inline-I2C sniffer perhaps? Or an interface board to inject i2c commands? Or something else?
Given the size of the part on here, and the limited I/O, it's not very useful outside of this use-case but works great here.
Oh my
It's one of those problems that throws you into a fit of self doubt and paranoia. You question your basic assumptions, your hardware, your colleagues, and God himself before making any headway and it turns out its often someone else's "fault", who programmed some core feature of a library 25 years ago in an obscure way, because they never could have imagined the Frankenstein-esque behemoth their code would be embedded in nearly 3 decades later.
>> (…) it’s a rare case of a kernel memory access check bug that had no security implications.
My favourite type of kernel bug.
Instead of worrying about simple and very common classes of bugs that can be solved statically with the help of the compiler, you are free to worry about whatever other non-trivial bugs in your program are remaining.
You're obviously free to waste your own time if you'd like.