PreInternet01

You might think that one would tire of unrealistic hot takes on the entire situation at some point (especially the 'Microsoft breakdown' in this one is grating), but I have to admit I still enjoy them, mostly since there are some promising opportunities here down the road that require the story to be kept alive.

First and foremost: bust mono-cultures in critical IT environments. Meaningful vendor diversity should be mandatory-by-law, and just like in the early days of the Internet, when nobody in their right mind would move into a data center that didn't have at least separate 'Cisco' and 'Juniper' uplink connectivity (so a vendor-specific bug wouldn't wipe you off the net: "ah, that unexpected BGP behavior took down all our peers for several hours" was, like, a weekly occurrence then), we shouldn't accept check-in-counter-workstations running exclusively on Windows today. It's already inexcusable that 'boot into the last OS snapshot that still worked' (which is commercial-off-the-shelf stuff used in most schools, just because 'little Johnny broke the PC again' is just a fact of life there) is apparently beyond oh-so-important mega-corps like Delta, but we need to go further.

So: mandatory hardware switches that select a boot into either Windows (running off one brand of SSD), or Linux or another completely-independent alternative (running on another brand of storage device, so that 'well, today is the day that all Seagate drives refuse to respond, due to an uptime integer overflow' is recoverable). One environment runs Chrome/Edge, Office365-connected-using-that-lovely-propietary-Exchange-protocol, the Microsoft TN3270 emulator and what-have-you. The other Firefox, OpenOffice with Thunderbird-over-IMAPS, Mocha TN3270 -- you get the idea. Run one environment for 3-to-4 days, then a mandatory switch-over, then back again, all year round. The training and making-everything-look-and-work-the-same development efforts will be fun, but nothing insurmountable.

Next up: Clownstrike and cohorts. Much has been made of the fact that custom kernel-mode drivers were used to track system activity, leading to this particular incident, but I can assure you, as a repeat victim of Enterprise Security Solutions, that even in plain old user mode, it's entirely feasible to mess things up beyond recovery using just the programming equivalent of a butter knife. And getting rid of layers like these will just never happen: "simply make the underlying OS secure enough" is not compatible with an open market for general computing, nor with the ambitions of Security Bureaucrats, and there will always be a need for 'zero day mitigation' which, when done reasonably well, can actually have dramatically good outcomes. I see this on a regular basis when figthing email phishing and spam: if you happen to catch a novel attack strategy and block it right away (which often requires an entirely new class of checks), you prevent a significant number of incidents. Snooze for an hour or two, and the (often painful) damage is done. So, "external vendors pushing cutting-edge code to all your devices on a real-time basis" is here to stay, and while certain players of course need to step up their basic validation game, regulation can only do so much.

So, what can be done: force Microsoft to ship https://learn.microsoft.com/en-us/sysinternals/downloads/sys... as a default and supported part of the OS, open-source it, have an industry-leading bug-bounty program, plus provide a rock-solid user-mode API that not only satisfies MS requirements, but also those of accredited third-party vendors. On Linux, where most interesting full-system observability tooling is still proprietary as well, find someone to open-source theirs (hey, Elastic has https://github.com/elastic/ebpf and some related stuff like https://github.com/open-telemetry/opentelemetry-ebpf-profile..., so perhaps that's a good basis?), with pretty much with the same requirements: rock-solid, open to reasonable criticism/improvements and a generally safe interface to whatever observability is deemed to be required.

TL;DR: the time has come for legally-mandated dual-ecosystem critical-use workstations with vendor-agnostic and safe observability APIs. First to deliver wins!

My concern is, as per my original post, that:

1. A single component in my app causes the same error over and over again, every second or so

2. This causes me to run out of my 10K, 50K, or whatever, fully paid-up 'Teams' or 'Business' allowance within the first day, and then my account gets disabled

3. Once my account gets disabled, my only option is to "contact Sales" and they never reply to me (in the case of Sentry), or quote me an amount that I'm simply unable to pay (in the case of many other similar services).

So, you could address this concern by stating that you now coalesce similar errors, and that my scenario is no longer an issue. Or, you could just punt me to a sales contact. Or, just, basically, tell me to "get gud" and get back in touch once my services have proper error handling.

Instead, you chose to pretend that I can't write properly, which, frankly re-confirms my notion about Sentry: RUN AWAY WHILE YOU CAN! This goes doubly for employees, BTW, so... consider yourself informed!

> Sentry's pricing plan has never gone from 10k errors straight to "negotiated enterprise deal"

OK, maybe not, but, on the current version of https://sentry.io/pricing/:

Developer (free): 5K errors

Team (US$312/user/year): 50K errors

Business (US$960/user/year): 50K errors

Enterprise (who knows): who knows

So, they now go from 50K (which sounds close enough to '10K 5-or-so years ago', right?) errors to "negotiated enterprise deal", and I have no idea whether that's worth it?

Do they coalesce errors these days? Again: I would love to use services like these, and I even don't mind getting a customized pricing plan, except that nobody like, ever, seems to want to talk about even the most basic details?

"Just give us your credit card number, and we'll see what happens, we'll make it work"... Right! But, but by any means, go and complain on HN that's it impossible to get paying customers...

So, since I'm an unrepentant cynic, I guess it's now, like 1.5 months until the inevitable "Sentry: it has been an interesting journey" blog post?

Anyway, so, I tried Sentry at some point, like, five years ago. Signed up for the 'team' plan, even though it's just lonely me, and had bugs coming in for about... a week, maybe less? Then, my subscription suddenly became inactive, because I had exceeded the 10K (or so, at the time) bugs that they were willing to track for me. Not, unique bugs, mind you, just the same bug that the same automated process was hitting every second or so.

But still, I now had to either negotiate an "Enterprise" price plan, or go away elsewhere. And I did actually ping them about the former, but never got a response. So, well, back to manually reading logs like an animal it is, then, I guess?

TL;DR: If your vision is to solve a certain problem, aren't the oh-we-didn't-see-that-coming corner cases the most interesting?

> Has it occurred to you that MongoDB exists?

My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""

So, care to take another guess? And, while we're here, does MongoDB run fully in-process these days? And/or allow easy pagination by ROWID?

> my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks

Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.

Practical example: I recently wrote my own SMTP server (bad idea!), mostly to be able to control spam (even worse idea! don't listen to me!). Initially, I thought I would be really interested in remote IPs, reverse DNS domains, and whatever was claimed in the (E)HLO.

So, I designed my initial database around those concepts. Turns out, after like half a million session records: I'm much more interested in things like the Azure tenant ID, the Google 'groups' ID, the HTML body tag fingerprint, and other data points.

Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.

Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent, but fortunately modern C# makes that easy as well.

And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)

> I miss that direct connection. The fast feedback. The lack of making grand plans.

There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated?

But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do.

Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff.

These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.

Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...

Yeah, also see: https://www.sciencedirect.com/science/article/pii/S001393511...

Spoiler: "Author presented a biased review about 7 potential effects of Wi-Fi exposure. Most of articles cited are in vitro or in animals and lab conditions, not in humans"

Oh, and to address the "why Windows" thing: because it works, generally speaking.

All the device drivers you need to talk to 8K displays, boarding pass printers, passport scanners, et cetera are actually there. Not necessarily good, but they're at least, like, available and sort-of tested.

Then, the "centralized IT management" story is actually quite decent for Windows. You have AD (no, sorry, Entra Ultra-Secure, or whatever it will be called next week), group policies, deployment kits, kiosk lock-down modes, controlled updates[1], and what-have you.

All of that is, ehm, sort-of lacking on other platforms. Actually, the worst that can happen to a well-meaning IT person is having to deploy and manage some proprietary "ultra-secure" Android abomination, especially after the single source of that goes inevitably bankrupt...

([1] Except, of course, when the outsourced elite security ninjas feel the need to chase down dangling pointers in kernel mode in realtime. They, of course, get to do whatever they want because, hey, National Security!)

Well, not everyone, it was just, like, 10% of all Windows installs.

But here's how things work from the perspective of a small-ish airport, healthcare, or finance operator (DMM is a D*mb*ss Middle Manager as employed by a regulator, WIP is the well-meaning IT person on said operator's end):

1. DMM: You're critical infrastructure! It's imperative that the Bad Guys don't take over your PCs!

2. WIP: Well, I'm pretty sure that our locked-down kiosks are pretty secure? We have separate VLANs without Internet access, basic anti-malware and basically only allow our TN3270 and digital signage stuff to run?

3. DMM: Oh, sweet summer child! You will connect to the Internet to share operational details with us and to show all-important public service notices at all times! And, you will employ an elite ninja strikeforce to keep Al Qaida out of all of that!

4. WIP: ehm, yeah, well, I'm pretty sure we can't actually afford any elite ninjas? I mean: we run, like, an entire nation from a run-down office in a low-wage country?

5. DMM: Nevermind, my lad! Here is a list of Approved Vendors that will Keep You Safe (and, most importantly, pay my humble consulting fee on the lowdown)

6: WIP: Err, that just looks like a badly implemented rootkit?

7: DMM: Well, it is what everyone runs to stay secure! And we do want to stay secure right? Let me talk to your CEO -- I see we're set for a round of golf tomorrow...

Well, go on, migrate your entire business to Apple, then!

You'll get forced to run Clownstrike on much shinier hardware, it will be fun!

But, by no means, step back, look at how you're running your business, and decide it's time to treat processes, never mind people, with the respect they deserve. Because the shareholders wouldn't have it, you know...

> After six months, the exec canceled the upgrade

This, combined with the otherwise non-specificity of the article gives me pause.

Is AI bunk? Sure! Even the latest-and-greatest models can't explain how to programmatically create a DNS record in Azure using the latest C# SDK, how to create a searchable/sortable React data-table with a remote data source, or how to achieve world peace (just to cite 2/3 topics everyone utterly failed at the last month when I tried).

Is this article bunk? Yeah, most likely -- no exec in any pharma company in their right mind would admit failure after only six mere months, and without any lawsuits...

OK, so here's my heartfelt plea: remove the 'unsafe' keyword from Rust?

Sure, not being able to do basic things like IO might be a bit of a limitation at first, but, that's all worth it, I guess?

Again: I'm pointing out to you that your absolutist stance on 'unsafe' and 'UB' is doing more harm than good.

You continue to choose to ignore this, which is your right. But as a "community leader" you could and should to better. As could I, I guess, by simply ignoring you, but, the mental health issues I see you cause in real-life make that sort-of hard...

OK, you truly seem not to understand how much damage you're dealing to the general population using absolutist statements like this, do you? Nor do you seem to understand "compromise", like at all, because you seem to equate it with "tit for that", which is unsurprising, but still... disappointing.

In any case, I'm truly done here, in all senses of the word, but I still I wish you and your acolytes the absolute best.

> UB is not possible in safe Rust, by design

You're available as an expert witness to that fact?

Because, eh, well, in at least one of the Rust-related situations that I'm involved in right now, someone might soon very well require the services of a person both as wise and reluctant-to-offer-any-kind-of-compromise as yourself...

Towards general mental health. I'm just a C# wage slave, and I'll admit, when being prompted, that my language, its vendor, its runtime environment, and its general approach are, to put it kindly, flawed.

However, as evidenced by the arguments and voting in this thread, Rust proponents will take no criticism, whatsoever.

I linked to a GitHub repository that documents many, many instances in which generally safe Rust causes UB.

The same kind of UB that recently hit one of my coworkers, caused a 3-day outage and now (despite all my counseling to the contrary!) will burn them out permanently.

My only request: can you guys please back off just a little bit? Programming is already hard enough without the purity wars you're stoking all the time...

> No serious person claims that Rust solves every problem ever

No, but there are a lot of people claiming that Rust cannot ever have any problems.

Just look at this thread. I merely linked to MIRI, and am currently at, like, -10 just for that.

Lots of people claiming that it just applies to 'unsafe Rust': is that true or not?

Regardless of anything else: can you, as a Rust community leader, please state clearly: is UB in generally safe Rust possible or not?

Ah, a voice of sort-of sanity, at long last.

So, the reason I posted my original reply, is that at one of my $DAYJOBs, we recently had a 3-day outage on some service, related to Rust. Something like using AVX to read, like, up to 7 bytes too many from an array.

Nothing really major -- we have a 10-day backup window, and the damage was limited to 4 days, so we were able to identify and fix all identified cases. But the person-to-Git-blame for this issue happened to be one of my mentees, and... they were blown away by it.

As in: literally heartbroken. Unable to talk about it. "But the compiler said it was okay!", crying. One of my coworkers pointed at MIRI, which correctly warned about the issue-at-hand, at which point I recommended incorporating that tool into the build pipeline, as well as (the usual advice in cases such as this) improving unit tests and focusing on X-1 and X+1 cases that might be problematic.

To this day, I'm truly worried about my mentee. I'm just a C# wagie, and I fully accept that my code, my language, my compiler, and my runtime environment are all shit.

But, as evidenced by my experience and supported by the voting in this thread, it seems that Rust users seem to self-identify with the absolute infallibility of anything relate to the language, and react quite violently and self-destructively to any evidence to the contrary.

As a community leader, do you see any room for improvement there? And if not, what would it take to convince you?

> What on Earth do you mean?

That documented use of safe Rust can easily lead to UB, which this infernal 'internal compiler representation' demonstrates.

I'm not even sure what is even remotely confusing about that?

Well, the general 'Rewrite All in Rust' consensus is that it solves all general programming problems, ever.

Yet, the linked repository shows a huge list of cases in which simple, documented use of Rust can cause Undefined Behavior (a.k.a. 'UB')

Pretty much every argument of Rust advocates against C/C++ boils down to either 'but memory safety' or 'but UB'.

Yet there are many convincing counter-arguments that boil down to 'but CompCert' or similar, and, as the linked repository shows, there might be at least some truth in there?

The one link for those who think that 'Rewrite it All in Rust' will, well, settle any debates: https://github.com/rust-lang/miri/

If you're the only author involved (or if all authors, including those of downstream dependencies, agree), you can dual-license the software: AGPLv3 by default, proprietary for paid-up customers.

So: whatever you put on GitHub (or wherever) is available for anyone to use for any purpose, as long as they re-contribute their changes to your code and/or the source code for directly-linked projects.

To anyone who doesn't want to meet those obligations, you can offer a proprietary license, allowing them to use the code in any way you see fit.

Turning this into a sustainable business, however, may be a challenge. So, some questions to consider: why do you want to open-source? Because you have existing dependencies? Existing contributors? Or just because it sounds good?

If the latter, just forget about it, and go full-proprietary: nobody will care. If you depend on contributors that might insist on open-sourcing their code, talk to them to figure out the best way forward. And if it's because of dependencies: you'll have to respect their license, and good luck with building a business on top of that...

Interestingly enough, accessing the Azure Portal from Firefox was OK, while Edge failed. I'm not sure what Edge's default DNS settings are, but after switching to 1.1.1.1 DoH, the portal also loaded.

That means the outage was either short-lived (doubtful, as it's not 9AM in Seattle yet, and Microsoft outages typically only start to get resolved around that time, weekdays only, mind you), or that... shockingly... it was DNS.

> And yet, since we moved to Denmark 3 years ago, every time we've returned to Denmark through Padborg, we've been met by border control

Sorry, but that's just business as usual. Document checks on trains have continued to be a thing despite Schengen since its inception, and are even common on commuter lines. I don't have any hard numbers on their effectiveness, but it's not unusual to see people being detained as a result (which is risky for the detaining agency, as mistakes are quite costly), so I guess there is a net positive there?

Note that, from a passenger PoV, these checks usually consist of "vaguely waving your ID card at the officers passing by" (i.e. less effort than having your ticket checked, which involves interaction with an RFID scanner...), which makes this ominous think-piece even less convincing.

To put this in perspective: for CO2 emissions (which is what we're talking about here, I guess?), you have the Big Four, in descending order of pollution: transportation, electricity, industry and other (mostly commercial/residential), for a total of about 65 Billion Metric Tons of CO2 per year.

In transportation, the biggest polluter (by direct emissions) by far is shipping. This is about 8/65 (12%), compared to 1/65 (1.5%) for the much-maligned aviation sector, or the 4/65 (6%) for cars, that, for some reason, nobody likes to talk about.

Then, when it comes to electricity, data centers are really quite marginal: 0.4/65 (about 0.6%). And, like aviation, the sector is taking measures to Be Better: higher airco setpoints, mandatory purchase of "green" electricity, etc. etc. Unlike the shipping sector, which pretty much continues to burn raw dinosaurs just because they're based in a jurisdiction that allows, nay, encourages that.

So, if you want to get mad about emissions, start with things that actually matter, would be my take? I happen to be of the opinion that AI is bunk, and any energy invested in it is wasted, but that goes doubly or triply for the shipment of novelty hats...

Well, I'm not a CrowdStrike customer, so I'm not entitled to any gift cards anyway, and I'll refrain from asking snarky questions like "is that per organization, per affected PC, or per minute of wasted support time?"

Instead, let me offer the following, alternative snark: "If I were to share with you the secret of renaming your C-*.sys files to C-*.tmp prior to trying to ingest them, so that if you crash while doing so, you will not repeat that mistake right after rebooting, how many US$10 gift cards is that worth? Keeping in mind, of course, that is, like, 2 hours of parking where I live?"

Nah, this kind of agitprop is not what any OS needs to succeed.

If $whatever were the dominant platform today, the number of incompetently-coded-but-mandatory-in-many-sectors kernel extensions would be, best-case, exactly the same as it is for Windows today, with pretty much the same outcomes (i.e.: generally not much, with entertainingly spectacular outliers like, well, Crowdstrike).

For $whatever to be actually better in a meaningful sense of the word, it would need to:

-Have a market share that, if not equivalent to that of Windows today, at least makes ISVs pay attention. This is really, really hard: not because "M$ controls all the hardware vendors historically and forever", but because hardware is an extremely low-margin business, and the room for experimentation is minimal...

-Similarly: most OS sales are to enterprises, and they have certain control issues that need to be addressed. "All that Active Directory (or whatever Microsoft marketing chooses to call it this decade) does" is a good starting point. Yes, this "compromises user freedom", bot possibly not in the way you think...

-Finally: in order to lock down kernel-mode access, offering user-mode APIs that are both completely fail-safe as well well as fool-proof for 80%, if not better, of use cases is mandatory. The engineering effort required to achieve this would most likely look a lot like that involved with, say, AWS, Azure or Google Cloud, with, say, similarly impressive results?

(2019), but still good-ish advice. The pattern works like a charm in modern C# as well, and has nice space-saving effects too by allowing you to omit the explicit variable declaration:

    if(!Whatever.TryParse<Thingy>(input, out var output)) output = some-sane-default;

or:

    if(!Whatever.TryParse<Thingy>(input, out var output)) throw new ApplicationException($"Not a valid Thingy: {input}");

Protip: don't do the latter in your kernel-mode driver.

I've seen plenty of people getting tripped up by this, sometimes quite publicly (like the blogger who returned the Mikrotik switch they got for their 25Gb/s home internet connection because turning on NAT turned it into a noisy space heater...)

Marketing or not, it's a useful metric when comparing expected performance across the Mikrotik product line...