Hubris – A small operating system for deeply-embedded computer systems
oxide.computer
oxide.computer
<<There are a class of "ideal attractors" in engineering, concepts like "everything is an object," "homoiconicity," "purely functional," "pure capability system," etc. Engineers fall into orbit around these ideas quite easily. Systems that follow these principles often get useful properties out of the deal.
However, going too far in any of these directions is also a great way to find a deep reservoir of unsolved problems, which is part of why these are popular directions in academia.
In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features.>>
It is a shame that Arduino/AVR never bothered implementing support for the full C++ library. If the full power of C++ is available to the end user, then perhaps alternatives like MicroPython would be less attractive.
Just as an example, the WiFi setup resembles a server far more than an ESP32 with esp-idf. All you do is give it the connection details, and MicroPython seems to handle the details like trying to reconnect in the background. It's not far off from what systemd-networkd or similar provides. esp-idf forces you to handle that yourself, and to think about what you want to happen in that situation.
MicroPython also doesn't support threads afaict, so you don't even have to handle scheduling threads.
I like MicroPython as a way to run Raspberry Pi like stuff on the cheap, and it's a great learning tool in that sense, but you're still too far from the hardware to really be learning about embedded systems.
Or like C64, Atari, ZX, Amiga, TI BASIC?
There is a REPL experience, and Python comes with batteries, even if MicroPython ones are tinier.
Python is the new BASIC in such hardware.
Also the Arduino folks don't seem to have that high opinion about C++, https://www.youtube.com/watch?v=KQYl6th8AKE
They only picked C++, because C would be even worse, and it provided an easy way to have their Arduino like language without creating their own compiler.
They have no plans to ever provide proper C++ support.
So if that makes you jump, go watch a Dan Saks talk about C++ adoption on embedded domain or not.
The actual runtime framework with its “setup” and “loop” methods is a reasonable proxy for an RTOS or the framework an experienced embedded developer would have built as a general runtime system.
There’s nothing wrong with Arduino, except that its SPI SD card library won’t give you good bandwidth but that’s because they wanted it to be an understandable simple access library for SD cards, and you will need to go further if you want reasonable performance.
Once you started with BASIC you presumably moved to learning assembly language, as many of these machines gained c compilers, you might have tried to obtain one.
The entire point of micropython is a friendly introduction or friendly prototype platform or learning platform. In no way does micro python take advantage of the hardware nor could it ever directly talk to hardware.
One should not treat all programming languages the same as they have different purposes and python is not fit for the purposes a c or c++ is fit for, aka memory allocation etc. The number one lesson a beginning embedded programmer should take away from arduino is that controlling hardware is about writing specific bit patterns to memory locations. Sorry, this is not something python can do or was designed for.
Deeply embedded means “embedded Linux won’t suffice”
Your car braking system had better not be a micropython program.
There are actual safety proofing systems in which code is proven, and the python interpreter itself will not come close to passing as the complexity is too large. (Formal verification is the Search term you seek (
I would have guessed quite a lot of people went from BASIC to Turbo Pascal. But you're talking 8-bit machines; maybe that was only available for 16-bit and up?
The jump from this to RTOS is large, though. The abstractions are different. The limitations are different. You probably need to rewrite anything you need. And what do you gain? Mostly only predictable latency, and the ability to run on very limited (but cheap) hardware. Which you need why?
If I'm selling a million Tamagotchis I'd rather use the 5 cent part over the 50 cent one and pocket the extra $.45M. A $5 part is a nonstarter.
Maybe in theory, but in practice most people still ship an entire OS in their containers (most of the time it's Alpine and it's not to big of a deal, but too many times it's an entire Debian!)
Datalog's incompleteness for example allows it to resolve queries faster than a complete language like Prolog due to the simplifying inferences it can make about the code.
With IPC, latency becomes the elephant in the room. An RTOS can't remove that, but it can help.
It is already more common than common monolith defenders think.
Huh, this is more like pragmatism than "Hubris".
At least originally; then the thing so named may drift away from what it originally was (intended to be).
Or sometimes the name is self-depreciatingly ironic / sarcastic, which I'd guess was the case here. So kind of "descriptive-but-in-reverse".
- Hubris for deep embedded
- Redox OS for Desktop/Server (https://www.redox-os.org/)
- Tock for embedded (https://www.tockos.org/)
- Xous for trusted devices (https://xobs.io/announcing-xous-the-betrusted-operating-syst...)
I assume there are more.
Even if it is completely unrelated, it is 'at least' production ready and will be used more than these projects would ever be used in terms of Rust projects.
There is an implication here that Rust will help make smart contracts "secure", but AFAIK the vulnerabilities in smart contracts have been in their logic, not in their memory/type safety or whathaveyou.
Are there even any cryptocurrencies that allowed less-safe languages like C in the first place?
In my opinion (as a Rust dev), Rust is weirdly over complicated for what they’re trying to do. Common types scripting languages are basically more than sufficient (and safe)for these applications.
It’s only a matter of time before a crypto project overplays the safety of Rust and them has a huge heist due to a logic bug, which will further contribute to jokes about Rust programmers. Most of the Rust devs I know are wary of Rust crypto projects.
Given the context assuming smart contracts you may have a point. Often consensus will be much slower and a limiting factor in execution. Solana may be a bit different here in their efforts to parallelise independent transactions.
High level provable languages always seemed like a good idea to me for smart contracts. As you say Rust doesn't necessarily seem like the sweetspot for this.
The Ethereum EVM assmebly is wildly unsafe but regularly used for the sake of gas and other things that would be impossible (most hilariously string manipulation). Solidity is unsafe with respect to things like overflow. It doesn't have memory unsafety in the traditional sense. Partly because it is allocate only and you don't have to deal with bounds checking yourself.
In one side of the fence you have Azure IoT Edge, done in a mix of .NET and Rust.
https://msrc-blog.microsoft.com/2019/09/30/building-the-azur...
On the other side you have the Azure Sphere OS with a marketing story about secure IoT, yet the SDK is C only,
https://docs.microsoft.com/en-us/azure-sphere/app-developmen...
To the point they did a blog post trying to defend their use of C,
https://techcommunity.microsoft.com/t5/internet-of-things-bl...
I do a lot of Rust work, but C still occupies a prominent place in my toolkit.
In fact, this is fairly standard among all of the professionals I know. The idea that companies need to abandon C and only do Rust as soon as possible is more of an idealistic idea on internet communities.
Additionally they could also allow for all GCC languages like Ada and C++, given it is the SDK toolchain, instead of being C only, if the security marketing from Azure Sphere is to be taken seriously.
This is pretty famously not limited to Microsoft's views on Rust. Probably at any sufficiently large company, though Microsoft is the tech company that gets most discussed about. At a previous employer, there was a division that was gung ho on Go and React, while another was Java and Ember.
Microsoft Security advises 1. managed languages, 2. Rust, 3. C++ with Core Guidelines.
Then comes out this group selling "secure IoT" devices with a C only SDK.
It's only "schizophrenic" to the degree that you expect an organization of thousands of individuals to behave like a single human mind.
Given your description apparently on US Gov anyone can get away with not following them, no wonder an US company acts accordingly.
It has a similar set of usage idioms to Hubris it looks like in terms of trying to setup as much as possible ahead of time to assemble what's kind of an application specific operating system where everything your use case needs is assembled at build-time as a bunch of communicating tasks running on seL4.
We recently added a concise little persistence interface that pulls in TicKV (https://docs.tockos.org/tickv/index.html) from the Tock project you referenced above, and some provisions are being added for some more dynamic task handling based on some asks from an automotive OEM.
Also, sincerest h/t to the Tock folks.
I defiantly think all the embedded Rust and even others will end up sharing lots of code.
I would like to spend some time working Datomic style on a more powerful immutable DB that could run on different KV interfaces. Lots of configuration should more immutable.
So how does a car manufacturer uses Rust and eventually be compliant with AUTOSAR/MISRA certification?
It would be interesting to know if there is some movement in that front.
MISRA and AUTOSAR are both consortiums in the automotive ecosystem. They're both industry standards bodies, not regulatory standards bodies (more like Khronos than ISO). The former worries mostly about defining best practices for convention and code style for primarily C and C++. The latter worries about defining a spec and interfaces for a middleware and application framework that's supposed to help automakers have a common portable (between MCUs) mechanism for developing, aggregating, and deploying ECU software. What's been merged from AUTOSAR to MISRA is basically AUTOSAR's codification of C++ subset and best practices into MISRA's version of C++ subset and best practices.
In the automotive world, the certification standard that ultimately matters is ISO 26262. It would be possible to write MISRA compliant code that still didn't qualify under ISO 26262 because of the process by which that code was developed. Likewise, it would be possible to have an AUTOSAR spec implementation that didn't qualify under ISO 26262 for similar reasons. It's also entirely possible to deploy software in cars that is neither MISRA compliant nor running on AUTOSAR. This is because ISO 26262 is 99% about process requirements and only about 1% technology prescription. Additionally, ISO 26262 is a self-certification regime almost everywhere in the world. It's a liability mitigation for when things eventually go wrong, not necessarily a direct regulatory hurdle that must be cleared. Lastly, ISO 26262 provides for the ability to offer evidence in support of deviations taken from the latest version of the standard's prescriptions and/or technology callouts.
Alright, now with all that out of the way...
With respect to seL4 and FerrOS. AUTOSAR Classic is for MCUs. seL4 and thus also FerrOS are not. They're for application processors. That said, it would be trivial to write FerrOS applications that could communicate over the protocols that are commonly defined for AUTOSAR Classic, so that you could have a FerrOS-built ECU interacting with the rest of your AUTOSAR-built ECUs. It would be likewise fairly trivial to create a basic HAL, carve out a slab of resources, and host an AUTOSAR Classic environment on one or more FerrOS tasks. It would just be kind of an odd thing to do given what kinds of things AUTOSAR Classic is used for and not entirely clear what the ultimate value would be. What I could imagine some day would be someone wanting to create an Adaptive AUTOSAR (a different AUTOSAR standard that's for higher-level applications and compute resources like ADAS and IVI) implementation atop something like seL4 via FerrOS (especially in the future when FerrOS supports bundling "applications" that aren't written in Rust).
As that relates to Rust in cars generally. That's a matter of risk tolerance and evidence investment. Rust has no ISO qualified toolchain, though Ferrous Systems has organized a project around a premise of getting Rust's development process, or some curated subset of it, to the point of producing the documentation and evidence of being so qualified. However, until that time it's incumbent on the user of the non-qualified toolchain to provide its certification evidence for the tools and technologies used. So, I expect for the companies who are dabbling in Rust, this is likely what they're planning on doing.
Do you know where I could find more info? Have I just been looking in the wrong places?
What is interesting about this OS except it is made with Rust? I mean some interesting architecture , new exiting features that are not in Windows/Unix world?
My question is if Redox OS is more similar to other hobby Oses we see here or is something more well thought like Plan9, Fuschia, Singularity ? Is there a link where someone that is not a Rust fanoboy (but maybe an OS fanboy) reviewed/described it in more detail?
Instead of everything being a file, everything is a scheme (basically a URL).
https://doc.redox-os.org/book/ch04-10-everything-is-a-url.ht...
There are some filesystem-like usage patterns built on top of that which are universal, but they're more limited.
1. In Hubris, all interrupt handlers dispatch to a software task. In RTIC, you can dispatch to a software task, but you can also run the code directly in the interrupt handler. RTIC is reliant on Cortex-M's NVIC for preemption, whereas Hubris can preempt in software (assuming it is implemented). This does increase the minimum effective interrupt latency in Hubris, and if not very carefully implemented, the jitter also.
2. Hubris compiles each task separately and then pastes the binaries together, presumably with a fancy linker script. RTIC can have everything in one source file and builds everything into one LTO'd blob. I see the Hubris method as mostly a downside (unless you want to integrate binary blobs, for example), but it might have been needed for:
3. Hubris supports Cortex-M memory protection regions. This is pretty neat and something that is mostly out of scope for RTIC (being built around primitives that allow shared memory, trying to map into the very limited number of MPU regions would be difficult at best). Of course, it's Rust, so in theory you wouldn't need the MPU protections, but if you have to run any sort of untrusted code this is definitely the winner.
Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions. I'm quite interested to look at the implementation when the source code is released.
Edit: I forgot to mention the biggest difference, which is that because tasks have separate stacks in Hubris, you can do blocking waits. RTIC may support async in the future but for now you must manually construct state machines.
The intent here seems to be that each binary has no need (and no ability) to get all up in another binary's business. Nothing shared, except access to the RTOS.
Alternatively I thought maybe you needed to have some closed source component from a vendor and could only include it like that.
Well, nearly anything having to do with wireless is typically blobs :/
Even Nordic has blobby SoftDevices, though you don't have to use them since Apache NimBLE exists (and rubble in Rust though that's only usable for advertising-only for now).
It's true that vendors are likely to be an area where we have to deal with some things being closed source, though we're not simply accepting that as a given. This is one reason we're writing so much of our own software, and also, poking at the bits that must remain closed: https://oxide.computer/blog/lpc55
Sounds like I should plan to do that on my own at the moment, but I'll keep it in a github style fork in case y'all's focus moves in that direction.
[1] https://rtic.rs/
What I did in a similar kernel was dynamically map them from a larger table on faults, sort of like you would with a soft fill TLB. When you turn off the MPU in supervisor mode you get a sane 'map everything' mapping, leaving all 8 entries to user code.
The way LDM/STM restart after faults is amenable to this model on the M series cores.
Now that the source is available, I took a look at what hubris does - it is not actually anything fancy, just a static list of up to 8 MPU regions per task [1].
It seems that leases aren't actually shared memory, but rather just grant permission for a memcpy-like syscall [2]. This is slightly better than plain message passing as the recipient gets to decide what memory it wants to access, but is still a memcpy.
[1] https://github.com/oxidecomputer/hubris/blob/8833cc1dcfdbf10...
I worked briefly at John Deere, and their home-grown operating system (called "JDOS", written in C) also baked every application into the system at compile time. This was my only embedded experience, but I assumed this was somewhat common for embedded operating systems?
> Precedence for the Hubris approach can be found in other systems like library operating systems, but there is an essential difference: Hubris is a memory-protected system, with tasks, the kernel, and drivers all in disjoint protection domains.
It's great that the drivers run in a different memory domains. Still I'm not convinced this isn't that different from many RTOS'es. Zephyr RTOS which can do memory protected tasks and defines each task's stack statically.
I linked two speeches where he goes over this in a bit more detail, but I hope the presentation opens it up even more.
I thought that had considerable issue with scaling...
https://youtu.be/XZmGGAbHqa0?t=224
Men with carts scale perfectly. One guy can manage 1000 machines. 2 guys can manage 2000. Perfect scaling.
(For reference: a crash cart is a cart with monitor, keyboard, mouse, cabling for them, and possibly external drives to help you install base OS, used for when you have no remote management)
It sounds to me more like at the scale you are at you are no longer the person making sure that individual computers are still running, and so are forgetting that this job needs to be done.
You can think in this case of the final product as pretty big hypervisor cluster that is delivered complete. I'll admit more than once I'd kill for that kind of product, and I suspect that the price/performance ratio might be actually pretty good.
The operating system in this case is used for internal service processor bits (compare: Minix 3 on Intel ME, whatever was that proprietary RTOS on AMD's PSP, etc. etc) that help keep the whole thing running and ship shape.
None of those are a great situation for multitenant scenarios.
This doesn't have to be control plane only. It could also be IO subsystems.
I have been wondering if it will become a thing in some cloud hosting services as well. I guess we need to see their pricing.
Basically there are a huge range of scale levels where having hardware on-prem make sense financially.
Also, Bryan Cantrill has some sort of personal nitpick with modern servers basically being x86 PCs with a different form factor, and with the fact that in modern servers hardware and software do not cooperate at all (and in some occasion, hardware gets in the way of software).
I strongly doubt this is aimed at Amazon, Google, or Microsoft (hyperscalers). They all already have their own highly customized hardware and firmware. If that is their target I wish them luck. There’s no margin and a ton of competition in that space and as long as they’ve been working on this that feels like a pretty poor gamble.
What I believe this is actually targeting is small enterprise and up. A company that has dozens to thousands of servers. They’re willing to pay a premium for an easier go to market.
Indeed, that is not aimed at those hyperscalers.
Even TecentCloud and Alibaba are relatively new addition to the term once people discovered their scale. Although generally speaking Hyperscaler used by industry analyst still dont include these two when they use the term.
All ranges of business except very small turn up in those purchases.
* decreasing the active codebase by at least three orders of magnitude
* using no C-Code (Rust only?)
* most code is kernel independent and not privileged (e.g. drivers, task management, crash recovery)
Also: Administration is mostly done by rebooting components.
However they will likely ship with a something derived from Illumos and bhyve hypervisor. You can then provision VM threw the API (and likely integrated with tools like terraform or whatever). You will likely not interact directly with Illumos.
Its basically attempt to help people make running data-center easier.
That is some great naming
It's hard to get software organizations to do ambitious things like this, and it's impressive that this was done on a relatively short timescale. I think the industry could learn a lot from how this was managed.
But someone from Oxide would need to tell you exactly how many RFDs took to desing and implement Hubris.
[1] https://oxide.computer/blog/rfd-1-requests-for-discussion
To summarize, it didn't take long to get to the point where Hubris was clearly the direction -- and a working artifact was really essential for that.
- Start with a really good engineer getting frustrated with existing stuff - but frustrated in a targeted way, and over a long period of time, not just a few weeks of grumpiness.
- Let them loose for a few weeks to sketch an alternative.
- Pause, and then the hardest part - smell whether this is going in the right direction. This just takes good taste!
- Make a decisive cut - either it's not working, or Let's Do It!
I can think of four or five ambitious projects I've been on or around that have really worked well, and they all seem to have worked in this way. I don't think I realized this clearly until this comment thread - thank you.
But horribly broken for me (mobile firefox), with text cut off at borders and overlaid by images.
The OS is named Hubris. Building a new Operating System does take a lot of confidence.
The debugger is named Humility. It can be humbling to know your program is not working correctly and use a tool to discover how it is broken.
Impatience would be a great name for the task scheduler. (Because you want your task to run NOW!)
Laziness would be a great name for a hardware-based watchdog timer. (Because you keep on putting it off / resetting it until later.)
Compare: https://www.threevirtues.com/
The site is quite confusing, stating that registration and login are only required for speakers, but then there is no information regarding the talks beyond the description.
That said, I believe that they're putting them all up on Vimeo after the conference, so it won't be long until you can watch them.
Are kind of way very much looking forward to the talk and the presentation of the operating system btw.
(And yes it's MPL)
;-)
Since they aim to open source everything, there probably will be a way to use their management plane and stuff with a homelab eventually :)
As much as most of this is way over my head, I'm always fascinated to read about new ground-up work like this.
Not to be too pedantic here, but it's important to note that the absence of C code, while arguably a benefit overall, doesn't by itself guarantee anything with regards to safety/security...I suppose there's going to necessarily be at least some "unsafe" Rust and/or raw assembly instructions sprinkled throughout, but I can't yet see that myself (as of the time of writing this comment, the GitHub links are responding with 404). Nonetheless, it's always refreshing to see some good documentation and source code being provided for these kinds of things. Many companies in this space, even these days, sadly continue to live by some outdated values of hiding behind "security through obscurity", which is somehow championed (though using different words) as a benefit even to their own customers, so it's refreshing that others (Oxide among them) are really starting to take a different approach and making their software/firmware publicly available for inspection by anyone inclined to do so.
There is some unsafe Rust and some inline assembly, yes. I imagine a lot less than folks may think.
It's all too often I see some cargo cult-style declaration of "no more C; it's all Rust" as if that has somehow solved all problems and absolved the programmer of the responsibility for ensuring their code is otherwise safe and correct (granted, Rust does make this easier to do), and IMHO this just ends up doing a disservice both to those proclaimers and ultimately to Rust itself. To be clear, this is not a statement against Rust by any means but rather a complaint against the conduct of some of its practitioners.
With that being said, I feel I really also need to state here that I absolutely do not believe the above to be the case with the announcement here...Even to the contrary, I would say, as the name "Hubris" says it all. It's great to see Rust used in practice like this, and I look forward to seeing more of the details in the code itself!
Rust is all about making it easier to ensure safety and correctness, yeah! It's still a tough job, but significantly easier than C or C++.
Yes, I did use it. We wrote an operating system in it at an aerospace company.[1] It didn't work out well. Sort of OK language, weak compiler, not enough memory given the poor code generation. It was just too early to be doing that back in 1978-1982. We got the thing running, and it was used for a classified high-security application, once.
That said, there are far more differences than there are similarities, and it's a gross oversimplification to say -- or even imply -- that the work in Hubris somehow replicates (or is even anticipated by) KSOS. More generally, I find this disposition -- that a new technology is uninteresting because it was "done" decades ago -- to be generally incurious, dour, discouraging, and (as in this case) broadly wrong on the facts. We as a team have as much reverence for history as any you will ever find; it is not unreasonable to ask those who have lived that history to return the favor by opening their minds to new ideas and implementations -- even if they remind them of old ones.
One of the most unusual features is that the Modula 1 compiler computed stack size at compile time, so that stacks, too, were allocated at compile time. Recursion required declaring a limit on the maximum recursion depth.
It's interesting as a working example of how minimal you can go running on bare metal and stay entirely in a high level language. Few languages since have been specifically dedicated to such a minimal environment.
This is what you need for programming your toaster or door lock.
[1] https://www.sciencedirect.com/science/article/pii/S147466701...
This all seems very cool, and I badly want to poke at embedded stuff again, but I have whatever the opposite of a green thumb is for hardware. Advice would be appreciated ^_^
1: https://www.st.com/en/evaluation-tools/nucleo-h753zi.html
It is completely beyond me, however, why one of the sockets is mounted upside down relative to the other one.
<meta name="generator" content="Asciidoctor 2.0.16">
so I guess https://asciidoctor.org/
https://github.com/oxidecomputer/hubris/pull/272
(specifically https://github.com/oxidecomputer/hubris/pull/272/files#diff-... )
QNX was one the most impressive OS I've seen, especially for its time. From the full OS in a 1.44Mb floppy disk to restartable drivers, real-time, etc. IPC with messages is built in and most things ran in userland.
It ended in the hands of BlackBerry which is probably not the best home for it...
Edit: I googled out of curiosity, and despite being closed source, it seems to still be marketed by BlackBerry and is supposedly a market leader in embedded OSes! More than 20 years later, well deserved.
https://www.automotiveworld.com/news-releases/blackberry-qnx...
https://hubris.oxide.computer/reference
Looks amazing imo. Waiting for github code :D
https://docs.microsoft.com/en-us/windows/dev-environment/rus...
The hype machine is pushing it into some places it is far from mature enough for. The hype depends more upon bad-mouthing other technologies than its merits can justify.
This is not to discount actual merits, or to doubt the needed maturity will come, in time. And, embedded is certainly a place it is already mature enough for.
And often multicore in embedded is not SMP, rather MCUs with multiple cores often end up running independent programs / OS instances on the different cores.
Maybe a unikernel (if applicable) could be a good compromise?
i don’t need all the features of a complete OS. i only need a small set of features, and i want them to work in a specific way.
EDIT: blog post is up: https://oxide.computer/blog/hubris-and-humility and the GitHub should be open.
EDIT 2: The HN story now points to this blog post, thanks mods!
> NOTE: support for this machine is dropped in recent Libreboot releases. It will be re-added at a later date. For now, please use Libreboot 20160907 on this machine.
Guessing by that number, this means support was discontinued five years ago.
https://chromium.googlesource.com/chromiumos/platform/ec on the EC and GSC (Google Security Chip) side.
However, I can’t quite get over their policy of paying everyone the same salary of $175,000. ( https://news.ycombinator.com/item?id=26348836 ) I’d love to apply and work on these things, but I wouldn’t love the idea of sacrificing $xxx,000 per year for the privilege of building someone else’s startup.
Does anyone know if they have some variability in equity compensation at least? I’m no stranger to taking significant compensation in startup equity, but it would have to be significant enough to make up for the significant comp reduction relative to just about every other employer in these domains.
The difference between $175K and market rate compensation (which may be significantly higher right now, considering the job market and the skills they’re asking for) is captured entirely by the founders and investors. We shouldn’t be shaming people for expecting a higher portion of the value they create in a competitive market like this.
But the fixed salary method creates a lot of secondary problems for a company: It can become a revolving door as people join for quick experience and resume-building, but then leave as soon as they can get a higher paying job somewhere else.
Why wouldn't some % of this be captured by your teammates?
One of my favorite employers right out of college did almost exactly this same thing: Everyone gets paid the same (although they had a couple tiers) and a lot of talk about how we’re all equal.
I believed it, until the acquisition event and I realized that the founders and early team members literally had 100-1000X or more as much equity than I did. All of the talk about paying everyone the same suddenly seemed like a cruel joke.
Another 10% salary makes a much bigger difference when your finances are out of control. Once you are living below your means it moves your retirement date around. It's not enough to move your Fuck You, I'm Moving to Tuscany date by an appreciable amount.
Berenstain Bears moment, maybe!
Basically, it doesn’t make sense for a startup to be frugal with compensation right now.
Unless you interpret it as their way of hiring only promising junior candidates or remote workers from locations where $175K is a lot of money?
Regardless, it’s weird to pay everyone the same amount of money because you’re basically pretending experience doesn’t matter. This leads to more experienced people leaving for other companies where they can (easily) get paid more while the less experienced people won’t leave because it’s a boost over what they’d get at other places.
I’ve worked at places with HR-mandated salary caps before. The best people always leave because there’s no hope of moving up and there’s no real incentive to work any harder than anyone else earning the same amount (as long as you avoid getting fired).
[0] https://oxide.computer/blog/compensation-as-a-reflection-of-...
Maybe on US, over here certainly not.
Worked out as their boss quit in the interim and now they're the boss.