yvdriess

I would be fine with a chaotic bubbly mess of an outside presentation, if the libraries were more robust and foundational. At the moment the underlying code, when you scratch the surface, have the feel of things thrown together to be replaced at later date.

I bounced off of bubble tea not because of the aesthetics and the unhelpful naming, but because of the programming model: a MVC-architecture cribbed from the Elm language. Why? It completely takes over and rips apart my CLI structure. A CLI is not a DOM or System.Windows.Forms, MVC is scattering around logic and adding indirection layers needlessly.

I am still using huh? and vhs, but their libraries have the feel of looking really good in demo and in the provided examples, but break down quickly when coloring just outside those intended lines.

Everything that was old will become new again. Content/structural version control used to be a research field. Pharo still uses one afaik https://scg.unibe.ch/archive/papers/Nier13bMonticello.pdf

Yeah. It's telling that this story is about their discord channel, not Teams.

The problem is that there is no baseline for measuring GC overhead. You cannot turn it off, you can only replace and compare with different strategies. For example sbrk is technically a noop GC, but that also has overhead and impact because it will not compact objects and give you bad cache behavior. (It illustrates the OP's point that it is not enough to measure pauses, sbrk has no pauses but gets outperformed easily.)

You could stop collecting performance counters around GC phases, but you even if you are not measuring the CPU still runs through its instructions, causing the second order effects. And as you mentioned too-short-to-measure barriers and other bookkeeping overheads (updating ref counters etc) or simply the fact that some tag bits or object slots are reserved all impact performance.

There is a good write-up of the problem and a way to estimate the cost based on different GC strategies, as you suggested, here: https://arxiv.org/abs/2112.07880

The way I found to measure a no-GC baseline is to compare them in an accurate workload performance simulator. Mark all GC and allocator related code regions and have the simulator skip all those instructions. Critically that needs to be a simulator that does not deal with the functional simulation, but gets it's instructions from a functional simulator, emulator or PIN tool that does execute everything. It's laborious, not very fast and impractical for production work. But, it's the only way I found to answer a question like "What is the absolite overhead of memory management in Python?". (Answer: lower bound walltime sits around +25% avg, heavily depending on the pyperformance benchmark)

Mandatory mention of notable actor languages:

  - Erlang and Elexir
  - E
  - AmbientTalk

And they could be 0- or 1- indexed? :P

TSO? Did you mean tso.architecture.cpu?

I agree on the part of the opportunity cost and that given the transistor budgets of the time a simpler design would have served better.

I fundamentally disagree on putting the majority of the blame on the object memory model. The problem was that they were compounding the added complexity of the object model with a slew of other unnecessary complexities. They somehow did find the budget to put the first full IEEE floating point unit on the execution unit, implemented a massive[1] decoder and microcode for the bit-aligned 200+ instruction set and interprocess communication. The expensive lookups per instructions had everything to do with cutting caches and programmable registers, not any kind of overwhelming complexity to the address translation.

I strongly recommend checking the "Performance effects of architectural complexity in the Intel 432" paper by Colwell that I linked in the parent.

[1] die shots: https://oldbytes.space/@kenshirriff/110231910098167742

> > Linear virtual addresses were made to be backwards-compatible with tiny computers with linear physical addresses but without virtual memory.

> That is false. In the Intel World, we first had the iAPX 432, which was an object-capability design. To say it failed miserably is overselling its success by a good margin.

That's not refuting the point he's making. The mainframe-on-chip iAPX family (and Itanium after) died and had no heirs. The current popular CPU families are all descendents of the stopgap 8086 evolved from the tiny computer CPUs or ARM's straight up embedded CPU designs.

But I do agree with your point that a flat (global) virtual memory space is a lot nicer to program. In practice we've been fast moving away from that again though, the kernel has to struggle to keep up the illusion: NUCA, NUMA, CXL.mem, various mapped accelerator memories, etc.

Regarding the iAPX 432, I do want to set the record straight as I think you are insinuating that it failed because of its object memory design. The iAPX failed mostly because of it's abject performance characteristics, but that was in retrospect [1] not inherent to the object directory design. It lacked very simple look ahead mechanisms, no instruction or data caches, no registers and not even immediates. Performance did not seemed to be a top priority in the design, to paraphrase an architect. Additionally, the compiler team was not aligned and failed to deliver on time, which only compounded the performance problem.

  - [1] https://dl.acm.org/doi/10.1145/45059.214411

In cpu uarch design, sure, but that's outside the context of the discussion. There's nothing you can do to that C++ library you are optimizing that will impact performance due to register allocation/renaming.

Your critique applies to measuring one or a handful of instructions. In practice you count the number of cycles over million or billion instructions. CPI is very meaningful and it is the main throughput performance metric for CPU core architects.

Register moves do not really play a factor in performance, unless its to move to/from vector registers.

Yes and just like our bodies, that closed loop is cooled by a rack of evaporators on the roof.

He was an animator and it's the interesting kind of factoids we read these comment sections for. We'll allow it.

It's the overhead cost caused by trust breakdown. (tbf sometimes the timesheets are there for legal/tax reasons)

It doesn't need to prove that. It needs to produce plausible data that appeases either your direct or +1 manager.

You don't need a second screen to do your job, denied. - your people manager that has to sign off on purchases

This is a pointless pissing match. It took decades to develop EUV into a high volume tool with R&D effort from different regions that each now contribute to some key part of the tool and supply chain. ASML or Zeiss does not make it a EU technology, the mask and resist does not make it a Japanese technology and Applied Materials does make it a US technology.

Go is a bit more of a low level language compared to actor languages where the language enforces that programming model. I think the point of the slogan is that you want to make the shared memory access an implementation detail of your larger system.

Same,tThe Settlers 1 theme now immediately starts playing in my head for the rest of the day.

On the software side I can recommend https://github.com/DrTimothyAldenDavis/GraphBLAS It is hard to make a sparse linear algebra framework, but Tim Davis has been doing a great job collecting the various optimal algorithms I to a single framework that acts more like an algebra than a collection of kernels.

A note on the hardware part: it does not require NN-specific hardware akin to neuromorphic. Sparse compute oriented architectures already have been developer for other reasons, such as large scale graph analysis or inference. It will still require significant effort to use it to train large models, but it would not be starting from scratch.

> Unstructured sparsity cannot be implemented in hardware efficiently if you still want to do matrix multiplication.

Hard disagree. It certainly is a magnitude harder to design hardware for sp x sp MM, yes; it requires a paradigm shift to do sparse compute efficiently, but there are hardware architectures both in research and commercially available that do it efficiently. The same kind of architecture is needed to scale op graph compute. You see solutions at the smaller scale in FPGA and reconfigurable/dataflow accelerators, larger scale in Intel's PIUMA and Cerebras. I've been involved in co-design work of Graphblas on the software side and one of the aforementioned hardware platforms: the main issue with developing SpMSpM hardware lies more with the necessary capital and engineering investments being prioritized to current frontier AI model accelerators, not because of lack of proven results.

Sparse matrices can increase in dimension while keeping the same number of non-zeroes, that part is self evident. Sparse weights models can be trained, you probably are already aware of RigL and SRigL, there is similar other related work on unstructured and structured sparse training. You could argue that those adapt their algorithm to be executable on GPUs and that none are training at x100 or x1000 dimensions. Yes, that is the part that requires access to sparse compute hardware acceleration, which exists as prototypes [1] or are extremely expensive (Cerebras).

[1] https://dl.acm.org/doi/10.1109/MM.2023.3295848

What do you mean by work better here? If it's for better accuracy then no they are not better at the same weight dimensions.

The big thing is that sparse models allow you to train models with significantly larger dimensionality, blowing up the dimensions several orders of magnitudes. More dimensions leading to better results does not seem to be under a lot of contention, the open questions are more about quantifying that. It's simply not shown experimentally because the hardware is not there to train it.

There has been plenty of evidence over the year. I don't have my bibliography handy right now, but you can find them looking for sparse training or lottery ticket hypothesis papers.

The intuition is that ANNs make better predictions on high dimensional data, sparse weights can train the sparsity pattern as you train the weights, that the effective part of dense models are actually sparse (CFR pruning/sparsification research), and that dense models grow too much in compute complexity to further increase model dimension sizes.

Yes! I'de been advocating for it inside the industry for a decade, but it is an uphill battle. The researchers can't easily publish that kind of work (even Google researchers) because you don't have the hardware that can realistically train decently large models. The hardware companies don't want to take the risk a rethinking the architecture CPU or accelerator for sparse compute because there are no large existing customers.

I will absolutely try this one out, it looks a lot more useful than binding caps lock.

It pops up a surprising number of times.

Keysets or chorded keyboards have been around since The Mother of all Demos, but never really seemed to have caught on. It makes sense to me to put them at your feet to chord some common actions or modifiers.

It is fairly common for emacs users to bind Ctrl or Meta to caps lock for improved ergonomics. There's also a bunch of RSI sufferers that are using foot pedals, which actually makes a lot of sense.

I personally switched to emacs for more than just Lisp when I started developing early signs for RSI. Switching to a purely KB driven interface has saved my wrists.