mrlongroots

> Is this just a cost efficiency thing?

It's not entirely, but even that would be a justifiable reason. Tail behavior of all sorts matters a lot, sophisticated congestion control and load-balancing matters a lot. ML training is all about massive collectives: a single tail latency event in a NCCL collective means all GPUs in that group are idling until the last GPU makes it.

> It only takes like 1 core to terminate 200 Gb/s of reliable bytestream using a software protocol with no hardware offload over regular old 1500-byte MTU ethernet.

The conventional TCP/IP stack is a lot more than just 20GB/s of memcpy's with 200 GbE: there's a DMA into kernel buffers and then a copy into user memory, there's syscalls and interrupts and back and forth, there's segmentation and checksums and reassembly and retransmits, and overall a lot more work. RDMA eliminates all that.

> all you need is a parallel hardware crypto accelerator > all you need is a hardware copy/DMA engine

And when you add these and all the other requirements you get a modern RDMA network :).

The network is what kicks in when Moore's law recedes. Jensen Huang wants you to pretend that your 10,000 GPUs are one massive GPU: that only works if you have Nvlink/Infiniband or something in that league, and even then barely. And GOOG/MSFT/AMZN are too big and the datacenter fabric is too precious to be outsourced.

Yes, unfortunately even the best intentioned individuals have very limited ability to make meaningful carbon-minimizing decisions. Carbon tax is such a sensible solution!

The other funny bit is that one-way PCIe latency is 250ns-ish (don't quote me on the exact numbers), which imposes a hard 1us constraint on latency between two hosts.

I think standard relational databases/schemas are underrated for when you need richness.

OTel or anything in that domain is fine when you have a distributed callgraph, which inference with tool calls does. I think the fallback layer if that doesn't work is just say Clickhouse.

Our social contract bestows individuals with freedom and governments with the monopoly of violence to police the limits of individual freedom.

If the acts of law abiding individuals (or groups) are a net negative for society, that is not an individual failure. Fiduciary responsibility is a useful parallel: it is not the job of a sugar manufacturer to think about the public health aspects of sugar. Their responsibility to their shareholders is to produce clean, safe, and edible sugar at competitive prices and do a good job with marketing and distributin, that's all.

If a selection mechanism is orthogonal to a property, it seems weird to argue that the selected subset is distributed differently along that axis than the broader population.

> "oh it's just database joins" is about like me ripping your arms off and describing it as "chemical reactions"

This argument is both inconsistent and counterproductive.

Inconsistent as in, the harm to me from having my arms being ripped off comes from you deciding to effect the intent to harm me. No photograph or x-ray of my arms can produce the intent of wanting to harm me.

Counterproductive as in, the "good vs bad" framing is pointless because it does not help with solutions. If your solution is to ban joins, you will have a hard time gaining traction for your cause. Strategic advocacy requires understanding axes along which you may be able to produce a coherent argument and gain leverage. "Ban joins" does not help.

> No doubt Deloitte or any other contractor shop would be able to do the same thing - but they don’t choose to.

I'm sorry but I absolutely disagree that the reason say Deloitte is leaving a few hundred billion dollars on the table is the presence of a moral compass.

Kitchen knives murder people. Toyota Hiluxes have powered more jihad than modern battle tanks. Our tastes, beliefs, and opinions as a society are shaped by recommendation algorithms run by facebook/instagram/twitter, to our profound detriment (personal opinion).

> And so its important for us to understand how they can be weaponized and to consider the social cost of that weaponization.

To be clear, I absolutely agree. Plenty of tech is double-edged. And Palantir very much so.

Let me restate my point. Palantir (or that class of tech products) is powerful at enabling visibility over a complex system. But visibility is not decisions, it is an input to decisions. If you had real-time telemetry from every single stomach, you could maybe automatically dispatch drones with food wherever someone is starving. Or you could use the data as a high-frequency indicator for a successful invasion. Morality is downstream of decisions not data.

I think Palantir is highly misunderstood.

As a technology, it is just database joins. It is just that they are able to pull in data from everything from S3 to SAP to ArcGIS, and provide analytics, visualization etc. on top to provide global visibility into any system.

The visibility can be "show me all illegal immigrant clusters" or "show me bottlenecks and cost sinks in CAHSR construction".

When we offload the moral impetus for society from politics to technology, we also squander control. Tech is tech and can be used for both good and bad. It is not that a strategy that aims to cap downsides by preventing the proliferation of technology is inherently bad, but it is doomed to fail. The evidence for dysfunction is not the existence of Palantir but in the failure of the watchdog layer of society (also called the government).

> The CPU can load from these virtual addresses directly into registers

This requires that device to bring meaningful amounts of its own memory. GPUs do that with VRAM. A storage device does not come with its own RAM, but interesting point!

From a response elsewhere in this thread (my current understanding, could be wrong):

> You can temporarily stream directly to the L3 cache with DDIO, but as it fills out the cache will get flushed back to the main memory anyway and you will ultimately be memory-bound. I don't think there is some way to do some non-temporal magic here that circumvents main memory entirely.

> This was a comparison of two methods of moving data from the VFS to application memory. Depending on cache status this would run the whole gambit of mapping existing memory pages, kernel to userspace memory copies, and actual disk access.

Yes, I think maybe a reasonable statement is that a benchmark is supposed to isolate a meaningful effect. This benchmark was not set up correctly to isolate a meaningful effect IMO.

> Also, while we’re being annoyingly technical, a lot of server CPUs can DMA straight to the L3 cache so your proof of impossibility is not correct.

Interesting, didn't know that, thanks!

I think this does not invalidate the point though. You can temporarily stream directly to the L3 cache with DDIO, but as it fills out the cache will get flushed back to the main memory anyway and you will ultimately be memory-bound. I don't think there is some way to do some non-temporal magic here that circumvents main memory entirely.

Agree: flat/relational structure is highly efficient to store, query planning/optimization/scaling become a lot easier, the language is clunky but relational algebra is beautiful, but...

I think pushing multiple joins to what is still a relational database and getting a complex output isn't the worst idea in the world, as a higher-level layer on top of a regular database.

On the other hand, "it needs four queries/RTTs" is not the worst thing in the world. It needn't be the goal of a system to achieve theoretical minimum performance for everything.

Let those who truly have the problem in prod push the first patch.

You don't need hugepages for basic 5GB/s sequential scans. I don't know the exact circumstances that would cause TLB pressure, but this is not it.

You can maybe reduce the number of page faults, but you can do that by walking the mapped address space once before the actual benchmark too.

Yes, it doesn't take a benchmark to find out that storage can not be faster than memory.

Even if you had a million SSDs and somehow were able to connect them to a single machine somehow, you would not outperform memory, because the data needs to be read into memory first, and can only then be processed by the CPU.

Basic `perf stat` and minor/major faults should be a first-line diagnostic.

Another thought: this is not meant as criticism or anything, just an aspect of performance that I thought was interesting but not covered by the blog.

A test that would show PG's strengths over ClickHouse for OLTP would be a stress test with a long-running set of updates.

ClickHouse maintains updates as uncompacted patches merged in the background, which is how you would do it with a columnar store. But if you have an update-heavy workload, these patches would accumulate and your query performance would start to suffer. PG on the other hand completes all update work inline, and wouldn't get degrading performance under update-heavy regimes.

This is just a fundamental artifact of OLAP vs OLTP, maybe OLAP can be optimized to the point where it doesn't really matter for most workloads, but a theoretical edge remains with row-based stores and updates.

Yep! A short summary of my n=1 findings (currently very speculative, but also sound I think, am meeting with my cardiologist next week to see what he thinks).

The type of ADHD I have seems to have an "autonomic nervous system impairment" component and a symptom profile overlapping with hyperadrenergic POTS.

1. I respond much better to Guanfacine ER (GFC) than stimulants alone (currently complementing with Vyvanse (LDX) 40mg, but I'd rate the Guanfacine as critical)

2. My blood pressure is very volatile, and GFC is supposed to have an impact but did not in my case, at least initially. I'd take GFC at bedtime and LDX in the morning, and on ChatGPT's suggestion, I asked my psych if I could take them together in the morning. Gamechanger for my blood pressure: the explanation seems to be that LDX makes my sympathetic nervous system extra simulated (on top of a poor baseline), and co-timed GFC balances it out.

3. I have poor cardiac endurance, and I find running nearly impossible. I'm a healthy young male who does weights and all. At ChatGPT's suggestion, I wore a Polar H10 and measured my resting heart rate while sitting, and then while standing still. I get a jump from 80bpm to 115bpm-ish, a strong indicator for something orthostatic.

I'm currently exploring rowing (with a concept2). I don't know why but it has a strong impact on my mental state that goes beyond general exercising: something about the rhythmic entrainment it produces, while being recumbent (good for POTS).

Ok I think the confusion arises because of the probabilistic nature of LLM responses that blurs the line between "intelligent vs not".

Let's take driving a car as an example, and a random decision generator as a lower bound on the intelligence of the driver.

- A professionally trained human, who is not fatigued or unhealthy or substance-impaired, rarely makes a mistake, and when they do, there are reasonable mitigating factors.

- ML models, OTOH, are very brittle and probabilistic. A model trained on blue tinted windshields may suffer a dramatic drop in performance if ran on yellow-tinted windshields.

Models are unpredictably probabilistic. They do not learn a complete world model, but the very specific conditions and circumstances of their training dataset.

They continue to get better, and you are able to induce a behavior similar to true intelligence more and more often. In your case, you are able to get them to stop and ask, but if they had the ability to do this reliably, they would not make mistakes as agents at all. Right now they resemble intelligence under a very specific light, and as the regimes under which they resemble one get bigger, they will get to AGIs. But we're not there yet.

A rebuttal is just an alias for "counterargument", it does not define the structure of the counterargument.

However flawed, what I said did have a structure (please refer to my other response in this thread for why).

> you have stated an assumption

I don't think I stated an assumption, this is an assertion, worded rhetorically. You are welcome to disagree with it and refute it, but its structural role is not that of an assumption.

"Can an LLM recognize the context deficit and frame the right questions to ask?"

> a bunch of non-sequitors

I'm guessing you're referring to the "canvas or not" bit? The sequitir there was that LLMs routinely fail to execute simple instructions for which they have all the context.

> not saying you are wrong

Happy to hear counterarguments of course, but I do not yet see an argument for why what I said was not structurally coherent as counterexamples, nor anything that weakens the specifics of what I said.

I have diagnosed ADHD and I agree largely with the person you are responding to.

The claim is not that ADHD is not a set of people with real psychiatric disorders, but it is a loose umbrella for what are actually disparate problems.

I recently learned that my symptoms, to a large extent, can be explained more accurately as POTS or something adjacent, and the meds I guided my psychiatrist towards were far more helpful than the stimulants I was being prescribed. This was a combination of me, reddit, and later LLMs arriving at me-specific diagnoses that go beyond clinical guideline regimes.

They could've just used `clock_gettime(CLOCK_MONOTONIC)`

> they will struggle just as a model does surly

A human will struggle, but they will recognize the things they need to know, and seek out people who may have the relevant information. If asked "how are things going" they will reliably be able to say "badly, I don't have anything I need".

I very much disagree. To attempt a proof by contradiction:

Let us assume that the author's premise is correct, and LLMs are plenty powerful given the right context. Can an LLM recognize the context deficit and frame the right questions to ask?

They can not: LLMs have no ability to understand when to stop and ask for directions. They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc. They can not even reliably execute my "ok modify this text in canvas" vs "leave canvas alone, provide suggestions in chat, apply an edit once approved" instructions.

It seems that most approaches treat neutron diffusion as memoryless (or at least stochastic):

https://en.wikipedia.org/wiki/Neutron_transport

> Consider the nuclear reaction metaphor. It's clearly not memoryless. Eventually you'll run out of fissile material.

I think no real process is memoryless: time passes/machines degrade/human behaviors evolve. It is always an approximation that is found/assumed to hold within the modeled timeframe.

It could, but important to keep in mind that the filesystem architecture there is also very different with a parallel filesystem with disaggregated data and metadata.

When you run `ls -l` you could potentially be enumerating a directory with one file per rank, or worse, one file per particle or something. You could try making the read fast, but I also think that it makes no sense to have that many files: you can do things to reduce the number of files on disk. Also many are trying to push for distributed object stores instead of parallel filesystems... fun space.

I think "making an unreliable system emulate a reliable one = bad" is too simplistic a heuristic.

We do this all the time with things like ECC and retransmissions and packet recovery. This intrinsically is not bad at all, the question is: what abstraction does this expose to the higher layer.

With TCP the abstraction we expect is "pretty robust but has tail latencies, do not use for automotive networks or avionics" and that works out well. The right question IMO is always "what kind of tail behaviors does this expose, and are the consumers of the abstraction prepared for them".

Even as a matter of safety/investigation hygiene, "pilot error" should be the conclusion of last resort, arrived at after months of poring over data, and because nothing else seems viable.

If we decided to pin all aviation incidents on pilot errors, we wouldn't even have invented checklists (what do you mean you forgot, try harder the next time).

"Natural" pilot errors lead to lessons that can be incorporated into design/best practices. That does not seem to be the case given current understanding: no flaw in any switch design seems apparent, and it does not sound like something you could do by accident.

So "pilot error" is not the "cracking the case"-grade conclusion it is being made out to be, it is an act of investigative resignation. In the days following the crash, allegations of mixing up flaps and landing gear were floated, and they all turned out to be wrong. This is not even accounting for the fact that the pilots are not around to plead their case, and basic human dignity requires us to defend their case until evidence clearly points a certain way