solatic

Yeah, I'm working on a communications platform as a side-project, architecturally providing reliable communications is exceedingly difficult to self-host.

* Mobile calls are another form of push notification, Apple/iOS requires setting up APNS and Google/Android requires FCM, there is no self-hosted option for that at all and, for battery life reasons, no independent replacement is supported. Genuine ownership / independence from the main project requires, iirc, basically compiling from source to register different IDs against APNS/FCM.

* Trying to get into telephony, integrating with SIP is a huge pain. Nobody wants to deal with this.

* Nobody supports high availability for the underlying calls. None of the cloud L7 load balancers support media protocols - you're dropping down to L3 UDP load balancing. All of the available solutions (including LiveKit) depend on stateful services that, at best, place ceilings on call lifetimes (e.g. 5 hours) to allow for graceful draining, but "calls" in Discord-style settings where people connect to a room and stay connected will easily outlast those ceilings. Not supporting high-availability, IMO, is a huge ask for self-hosters - the price isn't in the maintenance window itself, but in deferring updates until the maintenance window, which can leave you vulnerable, particularly if you decide to leave the firewall open to ingress from 0.0.0.0/0 for ease of use. And of course, as a self-hoster, you rarely have a full follow-the-sun ops team, so either you schedule maintenance when everybody else is off (and you should be off too) or when everybody is on (and it's disruptive).

You're right, but idTech is almost by definition that "novelty" kind of engine. And it did help id to sell more games. It's just apparently not enough.

This is not an argument that founders should seed 25k EUR to cover liability, it's an argument that GmbH bank account amounts should be visible publicly. If I want to put in a catering order, the catering business does not need 25k EUR to cover liability. If I want to build a data center, 25k EUR is not nearly enough.

> Surely having these guarantees for some packages beats having it for none? > You can just define that option in your own configuration

This is exactly what I'm talking about. Once you have Nix on your system, it infects everything. You want to let it control everything, because that is where its power is. Your life would be so much easier if you could let it control even more, like the stuff you share with your team. It's the siren call. And then you find yourself pitching it to your team, and then you inevitably fail to pitch it, because you ramped up to it over months and it's just impossible to get anybody else ramped up to Nix that quickly.

It's guaranteed heartbreak every time and I got sick of it.

> You can just silently use it and enjoy the convenience for the declarative parts of your setup, with no detriment to your ability to run the imperative, ad-hoc setup scripts that your company requires.

That's the kicker though. Nix's benefits come from the guarantees it can make based on its integration with the rest of the Nix-controlled ecosystem. Without the control, you don't get the guarantees, and you lose the raison d'être. You need to actively avoid the "value-added" parts (e.g. package options) because latest Homebrew upstream may give you a version that exposes an option that is not yet exposed by the package options, and you can't patch the package with Nix because you're not using a Nix-based package.

Chezmoi is declarative. The templates give me generated configuration. I can rollback anytime I want by reverting Git commits and calling chezmoi apply. It works well within its less-ambitious goals (compared to Nix).

I ran NixOS for a while, before I switched to Apple Silicon, so I consider myself fairly well-versed-enough (although nowhere near an expert) in Nix and the Nix ecosystem. My last four jobs have all issued me MacBook Pros; the last three with Apple Silicon.

Ultimately, my workplace setup is what has the most gravity. And the most I can get most workplaces to standardize on is Homebrew for package management of off-the-shelf software. Nix is so far outside of the wheelhouse for most engineers that I can't even propose it. It would be too much of a distraction for too many people for too long that it's just not seen as worth it and it's not worth spending the political capital on the attempt. Employers would literally prefer to run scripts from a whitewashing, barely-auditable Jenkins instance with parameterized jobs than to attempt to figure out how to distribute portable scripts and get everyone's permissions working.

So I need to pick software that will cooperate with other tools in an unstable fashion, rather than software that attempts to fully and exclusively control the environment to provide guarantees. Chezmoi fits. Nix and home-manager do not.

Necessary qualifier: for browser-based user sessions.

Plenty of good uses for JWTs for service-to-service communication.

edit: I read some of the linked stuff, e.g. https://paragonie.com/blog/2017/03/jwt-json-web-tokens-is-ba... . Please, if JWTs are such a horrifically insecure standard, go ahead and publish your means for hacking AWS STS's AssumeRoleWithWebIdentity , or don't publish and just exploit it by launching cryptominers in every Fortune 500 production AWS account. Let me know when you inevitably succeed, because JWTs are so insecure, right? /sarcasm

> many people who’d consider a VPS would happily slap a .env with an unencrypted secret then ssh to update

I just want to point out that you can totally still do this with Kubernetes. Of course it's not correct, but you can save that unencrypted secret in a .env file right into your container while you're building it - no need to use Kubernetes's support for supplying environment variables from the manifest. And of course, you don't even need a Dockerfile to build that container - you can just exec into a running container, paste it in, and then docker save.

Kubernetes doesn't save you from making stupid decisions, it just makes it easier to make better ones.

That's true, but it's about as helpful as a self-help book. Therapists, coaches, mentors are all helpful in that case.

> Have you considered just answering truthfully?

To give you just a little more context than other commenters -

You answer truthfully when you're interviewing from a position of power. Either you're already employed somewhere and you're taking your time exploring your options to see if maybe you can end up somewhere a little better, or you're an employer with applicants lined out the door and you want to winnow them down to the best match. In either case, you don't care too deeply if an individual interview sucks, you just move on.

Truth is always the first casualty of war. And when someone is out of work and fighting for their ~life~ livelihood, or a founder is trying to convince the first customer or the first engineer to take a risk on them so that they can get their baby off the ground, the truth dies real quickly.

Did you read the original article?

> The Facebook / Meta outage was so significant

The author specifically called out the Meta outage, as if he was offering a prescription ("It's easy to configure systems with tools like Ansible or pyinfra at scale") that would have prevented Meta (at Meta's scale) from suffering an outage. The argument that Meta should not have used DNS except that Meta runs at a scale where DNS is necessary... who comes up with these arguments?

It replaces DNS's pull-based architecture (contact a DNS server to get the IP address) with a push-based one (push the IP addresses to each /etc/hosts file).

Suggesting that a push-based, Ansible-based architecture will scale to hundreds of thousands of targets, with such pushes happening hundreds if not thousands of times a day, is a junior-level idea at best, dark comedy if I'm being charitable, and professional malpractice at worst.

Use Ansible to update /etc/hosts on hundreds of thousands of hosts every time a host is added or removed?

Thanks for the laugh...

> Stocks going down didn't un-research drugs

Drugs cost pennies to manufacture after they are researched and make their way through the approval pipeline. There are many generic drug manufacturers who can work off the existing formulas.

The more apt comparison is that LLMs won't be un-trained. Opus 4.8 now exists. Even if Anthropic somehow went bankrupt, that particular asset could, at the very least, be sold for proverbial pennies on the dollar to a "generic" inference provider.

> Concentrated ownership of the wealthy is not synonymous with “the public.”

I 100% agree. It's terrible that large private companies like Cargill and any others large enough to make this list: https://www.forbes.com/lists/top-private-companies/ are allowed to continue operating without being forced to list on stock markets and being subject to public reporting requirements. There's a very big difference between concentrated ownership by the wealthy and public ownership.

> You are very literally arguing for plutocracy over democracy

You can only assume that public markets results in plutocracy if individual investors hold outsize amounts of outstanding shares. The elections for Boards of Directors are not competitive when an individual shareholder holds majority voting power. There is a separate argument for a wealth tax that I agree with, and there are arguments to be made for maximum share prices (reaching the maximum triggers an automatic stock split) to guarantee financial accessibility (cough, BRK.A, cough).

Then perhaps the big AI companies aren't actually worth that much.

The public should own more than half. Via the stock market. Where public shareholders can vote in elections to control the Board of Directors, and elect Directors that act in the fiduciary interests of shareholders, and return excess capital to shareholders by issuing dividends. Where any member of the public can decide to buy or sell shares, being the most important development in the democratization of wealth development in all of human history, second only to the index fund that let members of the public put wealth development on autopilot?

When did public ownership mean that the government needed to be the owner? And when did we start to allow companies to float so few shares that public shareholder voting rights became largely meaningless?

The issue I have with these pieces is that AI will not affect the whole economy evenly. The disruptions come in bursts and fits: first digital artists were disrupted (e.g. Adobe Firefly), then junior engineering roles were disrupted, currently "measurer" roles (to quote Matthew Prince's piece on Cloudflare layoffs) are being disrupted, and it's rather foreseeable that occupations like lawyers and accountants are also at risk. But the key is that the disruption is not happening all at once. And that's a key fact because political disruption doesn't happen when a relative few people are laid off (like coal miners and factory workers were) but when a relative majority of people are hungry.

For such doomsdayer opinions to be correct, we'd see it in massive unemployment figures. US unemployment sitting at 4.3% does not bear that out. Finland and Spain are currently at >10+% unemployment. US youth unemployment may be at 9.5%, but British and Chinese youth unemployment are higher than 16%.

Is there some Finnish, Spanish, British, Chinese civil unrest that I'm not seeing in my media outlets?

> But then why is it structured in that manner? What's the purpose?

This is bizarre. When you agree to take a $100k base salary, you don't get all $100k on the first day; your salary is split into pay periods, and if you leave earlier (voluntary or not) then you don't get the rest of the year's salary by default (severance aside).

I'll agree with you that RSUs for public companies should not have cliffs. But the idea that you agree to a large amount of compensation up-front (so that re-negotiation is infrequent) which is then paid out in portions on a regular basis is very standard, for both cash and equity.

I don't think this is actually anything new. In large-enough companies, even before AI, it was and is quite common for executives to lose touch with base reality. I don't think anyone is under any delusion that people like Mark Zuckerberg intimately know the entirety of their corporate codebases. Everything is filtered through layers and layers of middle management whose summaries, cherry-picked statistics, and perpetually up-and-to-the-right graphs make it difficult to have an objectively informed opinion. Companies would, are, and will have mass layoffs that unintentionally (or, intentionally but with indifference to the consequences) fire key engineers whose loss results in "familiarity debt" within the systems those engineers owned.

Calling this "psychosis" is maybe a neologism but it's apt in perspective.

All that's actually new with "AI psychosis" is an acceleration of that phenomenon. The agents will summarize status faster than any middle manager. Claude will happily draw you any "up-and-to-the-right" graph you please, with the most common contemporary examples being "tokens burned" and "lines of code written". And vibe coding doesn't even require paying the cost of a mass layoff to get the "familiarity debt".

There have always been both good and bad engineering leaders. No tool will magically make a bad leader into a good leader overnight. There is nothing new under the sun.

Why does the voice need to be sent to the server? Why not perform speech-to-text on-device? Is the p10 phone/laptop not capable of this yet, despite every "dictation" feature I see in every modern OS?

> Code is a liability

You're over-simplifying. Code in and of itself is neither an asset nor a liability. The minimal amount of code needed to solve business needs with no additional complexity is an asset with some maintenance liabilities attached (same as how a farmer's tractor is an asset that needs to be maintained), with depreciation if unmaintained (bitrot). Any code used to build unnecessary complexity is pure liability.

Guess this is an unpopular opinion:

If you run more than one service/codebase, you might be better suited to using a proper container orchestration platform. Doesn't have to be Kubernetes. AWS ECS, GCP Cloud Run, Kamal are all modern options here.

If you run a single codebase in production, why are you even containerizing? Language ecosystems have done a phenomenal job of improving their dependency management since Docker was released. Python has uv. Go has modules. NodeJS has pnpm. Do you actually get benefit from containerizing if you're deploying to a single production host somewhere?

> people will choose fresh over canned, for obvious reasons

Not at all obvious. A lot of "fresh" produce in the US was refrigerated for more than a week before it arrived in the supermarket, from varieties that were designed to hold up to transport rather than flavor. Fruit that was canned at the height of the season is often much more flavorful than "fresh" off-season fruit.

The US has a problem with packing fruit in added sugar, which is sad but not inherent to canned fruit.

Not all alerts are created equal. You should generally have three levels of alerts: critical (which pages somebody, time-to-fix should be ASAP), warning (creates a ticket, time-to-fix should be within a few days), and suspicious (does not notify, appear only on an alert dashboard). The suspicious alerts are there to help guide your investigation on a critical or warning alert.

Each critical and warning alert should link to an "interactive runbook" - a dashboard that combines text instructions along with graphs showing real-time data.

Doing this at scale, correctly, requires both alerts-as-code and dashboards-as-code, which almost nobody does because nobody treats higher-level configuration languages (jsonnet, CUE...) with the attention and respect they deserve /cries-in-yaml

I think a spreadsheet is still helpful in this case.

> So you need to consider the emergency of moving somewhere in a political and legal climate not known in advance.

Fair enough. So do the financial planning, and ask yourself, how much does it cost to fly/travel to X place that is far away? Put in a risk premium - what if the cost became 2X or 3X because of a sudden catastrophe affecting everyone? What is the actual number that you need to save? Can you keep the money in a bank account, or are you concerned that banks will be inaccessible, and thus need something more portable? If you need something more portable, how much will it cost to protect it (vault/safe, weapons)?

I sympathize that life isn't fair, and that financial goals for some people need to be concerned first and foremost with personal safety instead of luxurious "nice-to-haves". But my point is that there are still actual numbers involved, and that you can put those numbers into a spreadsheet, and that the spreadsheet can help you understand your progress towards those goals. Financial planning is valuable regardless of what your goals are.

The most helpful tool here, I've found, is maintaining a personal finances spreadsheet.

It's one thing, without a spreadsheet, to have the emotion of "I'm not going to have enough and I need to save more to make sure I have enough." I'd argue it's even evolutionary; we want to make sure we have enough to get through the winter we know is coming.

It's quite another when your spreadsheet shows you saved X, your monthly cost of living is Y, and therefore you have enough money saved, even if your income went to zero and you made no changes to your lifestyle, to last you for Z years. Being able to take YouTube University rule-of-thumb advice like "of your take-home pay, use 65% for necessities, 15% for long-term savings, 20% for enjoyment" and seeing how much money that is per month for you in your personal circumstances, along with rules of thumb on things like what ratio you should have achieved by which age on net worth-to-income ratio (1x by 30, 2x by 35, 3x by 40, etc.) and seeing what your personal actual ratio is, to get a sense of benchmarking yourself.

I mean, the influencers could be totally full of shit, but it doesn't matter. Getting actual numbers for where you are, plus getting "generic" advice that you know wasn't directed at you personally, and seeing how those numbers make you feel, can do a lot to either tell you "you don't need to be so frugal anymore" or "yep, your emotions are totally justified, keep saving".

Azure's management APIs break connections coming from outside Azure's network every time they use DNS to execute a blue/green swap on their public load balancers. Existing connections are not gracefully drained. Terraform state gets corrupted (it thinks the operation failed when it actually succeeded and the resource was actually created) and requires manual fixing.

This happened frequently enough at large enough scale we seriously considered building an automation to attempt to analyze the Terraform logs for the connection breaking and automatically import the created resource.

Azure support was completely worthless.

Decisions rarely need to be made on-the-spot in synchronous meetings. You can have asynchronous approaches with shared documents and RFC processes, where you make everything available if people want to contribute to areas that they find interesting. This does not, of course, mean that decisions need to be made by committee, and people who provide feedback should understand that getting the privilege to provide input does not mean that they also get a veto.

It's quite rare to find companies that do this for the same reason it's quite rare to find companies willing to "do agile correctly" and really scope out work before sprints and not put additional work in the middle of a sprint. It takes too much effort and gives up too much flexibility for most managers to make the investment and see if it pays off.

I have no doubt that this approach works better than recurring meetings, but I do want to point out the trade-off, which is that this approach requires much more management attention and energy to keep their finger on the pulse and ensure all concerns are handled, compared to their time management being on autopilot with recurring meetings.

So it's a bit of a tautology. Managers who manage time better are more effective. Who knew?