6,784 karma · joined April 24, 2012
Otherwise, I keep it around for the desktop Excel app. Still my preferred spreadsheet app, even though Google Sheets does pretty much all of what I need.
Yup, this is the key issue and what makes it primarily a people problem. Technical solutions don't work if the main problem is getting buy-in to spec/build/adopt one, unless you're willing to build a lot of things you end up throwing out. So instead the bulk of the high risk work is actually negotiation between people.
There's a small but substantial number of engineers out there who haven't operated at the kinds of scales where hyperscalers' limits become normal architectural problems and don't have the humility to imagine that it could be the case. (e.g. blob stores do in fact have limits you can hit, and when you operate at petabyte scales you have to anticipate in the architecture that you can hit them for even trivial operations.) I also work on petabyte datastores and have encountered a bunch of those engineers over time.
To be fair though, that's the small minority of engineers I've encountered, and if it wasn't arguing about the types of scale problems unique to petabyte scales, it'd be about some other nuanced subject matter. It's a humility problem.
* Conway's law causing multiple different data science toolchains, different philosophies on model training, data handling, schema and protocol, data retention policies, etc.
* Coming up with tech solutions to try to mitigate the impact of multiple silos insisting on doing things their own way while also insisting that other silos do it their way because they need to access other silos' data.
And the reason standardization won't happen: the feudal lords of each of those branches of the hierarchy strongly believe their way is the only way that can meet their business/tech needs. As someone who gets to see all of those approaches - most of their approaches are both valid and flawed and often not in the way their leaders think. A few are "it's not going to work" levels of flawed as a result of an architect or leadership lacking operating experience.
So yeah, it might look like technical problems on the surface, but it's really people problems.
* You have a disability that hinders your ability to type on a keyboard, so you need extra time to type the essay based exam through vocal transcription.
* You broke your dominant hand (accidents happen) so even though you know all of the material, you just can't write fast enough within normal "reasonable" time limits.
* You are blind, you need some way to be able to read the questions in the test. People who can see normally shouldn't need those accommodations.
I don't think those are cases where you are lowering the bar. Not by more than you are allowing the test taker a fairer chance, anyway.
The problem is when you get into the gray area where it's not clear than an accommodation should be given.
For better or for worse, politics and randomization is just a thing in our jobs, and for at least some people in your team that means part of career growth is learning to handle it. If you, the manager, are the sole person capable of being the "shit umbrella" for the team, that's another way that your team gets a bus number of 1. (I learned this the hard way.)
In an ideal world you have some senior engineers who are more of the "don't bother me and let me cook" persuasion, and then you have at least one who is probably on track to become a manager, and they are your backup when you can't be in two places at once.
You could also argue that 100 out of 200 on a fair coin is more likely than any other specific outcome, such as 93/200, so if the argument is that the coin is "too perfect", you then also have to consider the possibility that the coin is somehow biased to produce 93/200 much more often than anything else, vs. 100/200.
I see why you'd think this, but I disagree. In my opinion it's two sides of the same coin, and the key moral question is whether you use those skills in a moral way. I've seen both well-meaning and charismatic people and not so well-meaning charismatic people, and at the end of the day I believe that charisma is a powerful tool, but it's not fundamentally good or bad.
Social interactions have always felt like a game whose rules I don't intuitively understand, and I've always envied people like my wife who handle it much more naturally and fluidly. The same way that I'm comfortable and capable in analytical settings, they navigate social settings with just as much finesse. I've personally spent a big chunk of my adult life trying to learn to navigate social interactions more comfortably and more intuitively, so I can see some parallels with what the author writes about. (For the record, I'm neurotypical, just awkward.)
For most people I don't think it's about charming, manipulating, or gaming social interactions, I think it's about wanting to make connections and friends because that leads to being happier.
100% agree when it works in your favor. We use it for exactly that situation where a non-zero false positive rate is fine and you can choose how much memory to devote to getting it closer to zero.
There have a been a couple times though where we've needed a "keep everything not on this list" operation, and unfortunately bloom filters don't work well for that situation. There are alternatives, but none as elegantly compact as bloom filters.
They are still "listeners", it's that events aren't pushed to the listener. Instead the API is designed for sheer throughput.
If you don't need a lot of perf but you place a premium on ergonomics and correctness, this sounds more like you want a workflow engine? https://github.com/meirwah/awesome-workflow-engines
Is this some scripting to automate your home, or are you trying to build some multi-tenant thing that you can sell?
If it's just scripting to automate your home, then you could probably get away with a single server and on-disk/in-memory queuing, maybe even sqlite, etc. Or you could use it as an opportunity to learn those technologies, but you don't really need them in your pipeline.
It's amazing how much performance you can get as long as the problem can fit onto a single node's RAM/SSD.
To be clear I wasn't talking about DB nodes, I was talking about skipping an explicit queue altogether.
But let's say you were asking about load balancing REST requests to different backend servers:
Yes, in the sense that "load balanced REST microservice with retry logic" is such a common pattern that is better understood by SWE's and SRE's everywhere.
No, in the sense that if you really did just need a distributed queue then your life would be simpler reusing a battle-tested implementation instead of reinventing that wheel.
1. Do you really need a queue? (Alternative: periodic polling of a DB)
2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process web service, or several instances, on one node.)
3. If it can't fit on one node, do you really need a distributed queue? (Alternative: good ol' load balancing and REST API's, maybe with async semantics and retry semantics)
4. If you really do need a distributed queue, then you may as well use a distributed queue, such as Kafka. Even if you take on the complexity of managing a Kafka cluster, the programming and performance semantics are simpler to reason about than trying to shoehorn a distributed queue onto a SQL DB.
Even though we keep an epipen around and we make sure they're not eating peanuts, we don't practice strict avoidance anymore, we don't have to worry about the "processed on shared equipment" warnings, and there's no problem if he touches/inhales/eats peanut, meaning we can eat peanuts around him.
The hygiene hypothesis is widely accepted, including by allergists, and there's definitely data supporting it, but we don't understand the mechanism, so it's hard to say that it's about any one specific thing vs. many contributing causes that correlate with hygiene and other aspects of the environment around the kid.
The advice about early exposure clearly works though, and there's data to support that early exposure even after confirming the allergy can increase the chances of outgrowing the allergy.
I agree and disagree here. IMO the sign of experience is when you understand which details really matter, and which details are more or less the same across the different stacks, and also when they don't know enough yet to differentiate and need to ask someone else.
How true this is. How much of our code, how much compute power, and how much human energy is spent entirely on transforming data from one encoding or schema to another, for the purposes of being able to apply some algorithm or human process to it? Sometimes I feel like it's at least half of our silicon, electricity, and human time spent just doing that transforming.
It really helps to front-load the controversial questions. That way, if the team/organization/leadership gets mired down in that discussion, it's (1) determining constraints to reduce your search space early, and (2) it isn't setting false expectations about how long the decision will take because the design work hasn't even started!
One thing I miss about working in a startup is that you didn't have that many stakeholders to go through, so there wasn't nearly as much communication/stakeholder overhead to work through.
(Context: I work at this scale)
Yes. However, as you can imagine, the processing costs can be potentially enormous. If your indexing/ordering/clustering strategy isn't set up well, a single query can easily end up costing you on the order of $1-$10 to do something as simple as "look for records containing this string".
My experiences line up with theirs: at the scale where you are moving petabytes of data, the best optimizations are, unsurprisingly, "touch as little data as few times as possible" and "move as little data as possible". Every time you have to serialize/de-serialize, and every time you have to perform disk/network I/O, you introduce a lot of performance cost and therefore overall cost to your wallet.
Naturally, this can put OTel directly at odds with efficiency because the OTel collector is an extra I/O and serialization hop. But then again, if you operate at the petabyte scale, the amount of money you save by throwing away a single hop can more than pay for an engineer whose only job is to write serializer/deserializer logic.
Take, for instance, Ford’s car from the 2003 FIA World Rally Championship. Through a knowledgeable automotive tuner and journalist named Stav from the UK (Facebook page), we learned that the 2003 Ford Focus RS WRC used an ingenious, mostly-hidden system to store excess pressurized air from the turbocharger in a titanium tank until it could be advantageously crammed into the engine on straight portions of the course, elevating power beyond what would otherwise have been possible.
English grammar is often confusing in cases like this because verbs mostly don’t have distinct spellings/pronunciations for the subjunctive mood, even though the subjunctive conjugations are part of the grammar.
It also doesn’t help that “subjunctive” is kind of the “else” condition among the three verb moods, the other two being “indicative” and “imperative,” so you often notice that it’s the subjunctive by ruling out the other two.