266 karma · joined April 1, 2023
I don't want this to become a game of "supplying the evidence" because that also becomes a game of the "skeptical party" DoSing the "earnest party". What I said was a best-effort distillation of my takeaways from years of following Indian news media, both free and paid.
I'm all for holding the government accountable, but the apt metaphor for some of the cited outlets would be that they protest for a road where the government hasn't built one, and protest for the tree where they chopped one to build a road.
> Volunteers are free to add.
I tried addressing this in my initial response. I have nothing more to add.
It's not that ANI is an absolutely non-partisan and an objective outlet. They do lean pro-government, but the yardstick being applied here is not consistent at all. No Indian news outlet is great by that yardstick, but one is being called an absolute sham, and those who consistently take anti-establishment stances, often without merit, barely get a footnote.
Now you could argue that Wikipedia is volunteer-driven, and you could submit an edit, but it is hard. During the farmers' protests ~3 years ago, articles were worded in a manner that led one to believe that deaths by natural causes among the protestors were somehow caused by the protests. I just checked the article as I was writing this response, and there is still a detailed section titled "fatalities" that mostly documents deaths from natural causes. I tried sending in edits for some of this back in the day but faced an uphill battle against other contributors and gave up because I had a day job to get to.
None of this justifies a page being blocked, especially outside Indian jurisdiction, but it would be unwise to ignore the broader context about the website being an ideological battleground and not being able to pull off the right balance.
Right! GPUs are bandwidth behemoths. If you have a 400 Gbit/s network and a GPU that can saturate it, there's a very good bandwidth reason to cut the CPU out of the path. For collectives, you are probably optimizing for latency vs bandwidth, but in all cases the less you move the data the better it is.
> Most topologies such as AllGather and AllReduce run with only 4 Queue Pairs. Not my area of expertise at all but wow that seems tiny! All this network, and basically everyone's talking to only a few peers? That's what it means right?
1. It seems that the cardinality of the collective groups is quite small (~128 GPUs), despite the reported cluster being 16k GPUs. They attribute it to "multidimensional parallelism" and while I don't speak much ML, I'm guessing that it's some sort of horizontal + vertical partitioning, and collectives are only triggered within a partition.
2. As these are collectives, all nodes do participate, but you create an overlay network with a ring or tree topology per collective. In a ring, every node just needs a QP each for left and right neighbors. For trees, it depends on the fanout. Now you could have shallow trees with larger fanouts, as latency is a function of the depth, and I feel like you can have more than 4 QPs (~500 is where NIC SRAM becomes an issue AFAIK). I think there is some buffering overhead per QP (beyond the standard NIC state) that they are trying to minimize, haven't read the paper thoroughly.
3. You do need a fancy network for collectives. They are a latency-sensitive operation on the critical path. The entire multimillion dollar thing is idle while an AllReduce is active, so 500us vs 100us in E2E latency matters a lot.
While on paper it is true that HPC has had collectives and dragonflies and RDMA for ages, a whole host of issues make tuning these things hard and tedious. Such work also informs the design of future "smart networks" by keeping everyone updated with common problems and workarounds that folks employ.
Finally I got some local electronics repair guy to just solder a new USB port onto the circuit board and that fixed everything. But never buying their phones again.
This is somewhat like AVX - it only works if you have large enough vectors (a simple operation that is repeated a million times on contiguous data). Arbitrary buildings are much harder to optimize.
Broadly, I think there's so much potential for computing in construction. If we can model logistics decisions as optimization decisions, we can leverage incredibly efficient solvers to extract efficiencies. The challenge, of course, being the "info pipeline" - integrating machines with construction processes and planning software and orchestrating it all in a way that adds value.
If an entire construction plan is modeled, you could also compile it into a staged procurement/shipment plan, and essentially orchestrate the supply chain in sync with the jobsite activities.
It'd be cool to have more info on how the process looks like with/without these machines, how rates of specific activities are impacted etc. but a bunch of this stuff is probably proprietary.