abc-1

Look at how mathematicians build minimal yet complete definitions for inspiration. An algebraic system can be created with a set of operations such as multiplication and addition, and existing concepts can be mapped to this system, such as money, but the underlying algebraic system will never change. It is complete.

Much of the system can be complete like this with forethought. The pieces that cannot can be factored out to the edges.

Contagion is exactly why interfaces are one of the most important pieces of design and should be given significant thought. A beautiful interface with a suboptimal implementation can be easily cleaned up when time is allotted. The reverse is rarely true.

Recursive self improvement

Nine women cannot give birth in one month.

And AlphaGo is trained to play Go? The point is training a model through self play to build neural network architectures. If it can play Go and architect chip placements, I don’t see why it couldn’t be trained to build novel ML architectures.

Why aren’t they using this technique to design better transformer architectures or completely novel machine learning architectures in general? Are plain or mostly plain transformers really peak? I find that hard to believe.

Nobody ever really believed this, the truth is rarely in vogue.

I do all of the above. Summary comments are incredibly helpful and have been validated by empirical research. Too bad many have drunk deeply from the Clean Code koolaid and can’t be saved.

Vector embeddings.

If you put a lot of arbitrary constraints to not allow it to be better, sure. Enjoy.

A lot of this reads like code search tools could and should be a lot better. They probably will be with AI finding its way into everything. In the old days, people would Hungarian prefix types, but now the IDE mitigates that with color codes.

ORMs do a great job at making easy things even easier and hard things a lot harder. If that sounds like a bad deal to you- it’s because it is!

Anyone know any research combining RNN like architecture to transformers? It’d be neat if a layer of a transformer could decide to loop its output vector to a previous layer n number of times to allow it to “think harder”. The implementation would be tricky to get right though.