Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things much easier to work with.
2,105 karma · joined September 25, 2022
> there was way too much academic interest in language concepts and way too little practical interest in doing work.
They are communicating something real, but perhaps misattributing the root cause.
The purely abstract ‘ideal’ form of software development is unconstrained by business requirements. In this abstraction, perfect software would be created to purely express an idea. Academia allows for this, and to a lesser extent some open source projects.
In the real world, the creation of software must always be subordinate to the goals of the business. The goals are the purpose, and the software is the means.
Languages that are academically interesting, unsurprisingly, attract a greater preponderance of academically minded individuals. Of these, only a percentage have the desire or ability to let go of the pure abstract, and instead focus on the business domain. So it inevitably creates a management challenge; not an insurmountable one, but a challenge.
Hence the simplified ‘these people won’t do the work!’.
It’s slowly, but inexorably increasing. The constraints are the normal constraints of a new technology; money, time, quality. Particularly money.
Still, token generation keeps going down in cost, making it possible to produce more and more content. Quality, and the ability to obfuscate origins, seems to be on a continual improve also. Anecdotally, I’m seeing a steady increase in the number of HN front page articles that turn out to be AI written.
I don’t know how far away the “botnet of spam AI content” is from becoming reality; however it would appear that the success of AI is tightly coupled with that eventuality.
So we need some mechanism to verify the content is from a human. If no privacy preserving technical solution can be found, then expect the non-privacy preserving to be the only model.
It’s nothing like being a carver. It’s like being a director; “Once more! With feeling!”. “Perfect, brilliant, just one more take, and I want you to consider…”
A sculptor shapes with their hands, and there is pleasure in that. A director shapes with someone else’s hands.
Which is ironic considering the subject matter. “Perfect”, but artificially constructed. “Just for me”, but algorithmic slop.
I agree that you can do so much more custom tailoring of bespoke software with the speed an LLM brings. But something inside of me still revolts at calling this anything other than “convenient”.
“Perfect” I will reserve for things I’ve made myself. However “imperfect” they may really be.
- There does not seem to be a way to determine which machines in the fleet have successfully applied. If you need a policy to be active before doing deployment of something (via a different method), or things break, what do you do?
- I’ve had far too many major incidents that were the result of unexpected interactions between group policy and production deployments.
I can’t go into details about current cases with my current employer, unfortunately. Ultimately, the requirements go through legal and are subject to back and forth with representatives of the government(s) in question. As I said, the problem isn’t passing an audit, it’s getting the initial approval to implement the solution by demonstrating how the requirement will be satisfied. Also, cloud companies are in the same boat, and aren’t certified for use as a result.
This is the extreme end of when you need to be able to say “x definitely happened” or “y definitely didn’t happen” It’s still a “log” from the applications perspective, but really more of a transactional record that has legal weight. And because you can’t lose it, you can’t send it out the “logging” pipe (which for performance is going to sit in a memory buffer for a bit, a local disk buffer for longer, and then get replicated somewhere central), you send it out a transactional pipe and wait for the ack.
Having a gov tell us “this audit log must survive a dc fire” is a bit unusual, but dealing with the general requirement “we need this data to survive a dc fire”, is just another Tuesday. An audit log is nothing special if you are thinking of it as “data”.
You’re a reliability engineer, have you never been asked to ensure data cannot be lost in the event of a catastrophe? Do you agree that this requires synchronous external replication?
Certainly you can solve for zero data loss (RPO=0) at the infrastructure level. It involves synchronously replicating that data to a separate physical location. If your threat model includes “fire in the dc”, reliable storage isn’t enough. To survive a site catastrophe with no data loss you must maintain a second, live copy (synchronous replication before ack) in another fault domain.
In practice, to my experience, this is done at the application level rather than trying to do so with infrastructure.
> There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies
It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must.
> Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all.
I care about matching the solution to the regulation; which varies considerably by country and use-case. However there are multiple cases I have been involved with where the stipulation was “you must prove you cannot lose this data, even in the case of a site-wide catastrophe”. That’s what RPO zero means. It’s DR, i.e., after a disaster. For nearly everything 15 minutes is good, if not great. Not always.
> In the world you describe, you don’t have any durability when the network is impaired.
Yes, the real world. If you want durability, a single physical machine is never enough.
This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally default to a DB (or cloud managed service).
The reason this matters for this logging scenario is that I normally don’t care if I lose a bit of logging in a catastrophic failure case. It’s not ideal, but I’m trading RPO for performance. However, when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.
But it isn’t! Because there are many hardware failure modes that mean that you aren’t getting your log back.
For the same reason that you need acks=all in Kafka for zero data loss, or synchronous_commit = remote_flush in PostgreSQL, you need to commit your audit log to more than the local disk!
That’s true. They specify that logs cannot be lost, available for x years, not modifiable, accessible only in y ways, cannot cross various boundaries/borders (depending on gov in question). Or bad things will happen to you (your company).
In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine”. You need to know that the record has been made durable (across whatever your durability mechanism is, for example a DB with HA + DR), before progressing to the next step. Depending on the stringency, RPO needs to be zero for audit, which is why I say it is a special case.
I don’t know anything about linux audit, I doubt it has any relevance to regulatory compliance.
In practice, the audit isn’t really a log, it’s something more akin to database record. The point is that you can’t filter your log stream for audit requirements.
<= 0 regrets
Unfortunately, educating yourself on this topic is not easy and involves differential equations. The economic models that fail to predict our current situation are simplifications. I’d link you, but I don’t think I’ll be getting a very receptive audience!
Puzzle no more, the answers are obvious! There are two interlinked mechanisms leading to this phenomenon. The rise of inequality (centralisation of power and wealth) and the rise in private debt. Both require coordinated governmental intervention to address, which won’t happen until the next economic crisis and dramatic drop in standards of living. Wish it was different, but economic theory (mainstream anyway) doesn’t account for our present situation and the control system is cycling into instability.
The upside is that we might learn the lessons this time around.
First of all, thank you for the honesty. It shows good character!
I think you are right that good character is the core of being a good manager. It’s the core of being a good person. Virtue and duty. Unfashionable words, but the secret to “happiness” (the good life). The ancient greeks understood this, and it’s been the heart of western philosophy.
We are all works in progress.
Sharepoint, onedrive, teams. Everything is integrated and collaborating is trivial. You can’t replicate that with open source.
This is an amusing response. I personally have chased the TC (to an extent), climbed the ladder, and sit at the top (below the c-suite). This isn’t cope, it’s a genuine reflection based on many years of experience having worked for multinationals in various countries.
Perhaps compensation should be a reflection of ability, but the real world is far more nuanced.
This is such a BS attitude. A person’s ability doesn’t map directly with their salary.
The very best developers I’ve ever worked with? Small companies outside of major cities (for life style reasons). Not everyone is trying to max their income; some want a good job doing something they love.
Small companies can offer more meaningful impact, if you’re not the type who likes being a tiny cog or commanding armies.
I don’t know how many forests we are burning to have a digital secretary, but surely the environment can take one more for the team?
Yes, it should be obvious. At least at the current state of LLMs.
> There's the long term incentive, but is there a short term one?
The short term incentive is keeping their job.
No technical debt is not the same thing as “perfection”. Good enough doesn’t mean perfect.
Would it be ok to submit an essay with only 90% of the underlined spelling mistakes fixed? Do you paint your outdoor table but leave the underside for later?
Do it once, do it right. That doesn’t mean perfect, it means not cutting corners.