GistNoesis

Thanks for pointing to the GGUF.

I just tried this GGUF with llama.cpp in its UD Q4_K_XL version on my custom agentic oritened task consisiting of wiki exploration and automatic database building ( https://github.com/GistNoesis/Shoggoth.db/ )

I noted a nice improvement over QWen3.5 in its ability to discover new creatures in the open ended searching task, but I've not quantified it yet with numbers. It also seems faster, at around 140 token/s compared to 100 token/s , but that's maybe due to some different configuration options.

Some little difference with QWen3.5 : to avoid crashes due to lack of memory in multimodal I had to pass --no-mmproj-offload to disable the gpu offload to convert the images to tokens otherwise it would crash for high resolutions images. I also used quantized kv store by passing -ctk q8_0 -ctv q8_0 and with a ctx-size 150000 it only need 23099 MiB of device memory which means no partial RAM offloading when I use a RTX 4090.

Programming is indeed becoming witchcraft, with LLMs it is of the utmost importance that you chose the right database administrator.

For example I'm now relying on Soteria, the greek goddess of safety, salvation and preservation from harm to act as my database administrator.

Who should get jailed ?

Does the company which willingly sells the polymorphic virus editor bear any responsibility, or should the unaware vibe coder be incumbent ?

There is also the fact that it's very easy to plant backdoors in LLMs with plausible deniability :

- You can just use the same tools you use to train them to make them behave in some specific ways if some specific preconditions are met.

- You can also poison the training data, so that the LLMs are writing flawed code they are convinced is right because they saw it on some obscure blog but in fact it had some subtle flaw you planted.

- You can poison the prompts as they are automatically injected from "skills" found online.

You couple that with long running agents which may drift very var from the conditions where they were tested during the safety tests.

You add the fact that in this AI race war, there is some premium to run agents capable of advanced offensive security with full permission, pushed using yolo dark-pattern.

The training process is obscure and expensive so only really doable by big actors non replicable and non verifiable.

And of course, now safe developers (aka those not taking the insane risk of running what really is and should be called malware), can't get jobs, get no visibility for any of their work, drown into a sea of AI slop made using a prompt and a credit card, and therefore they must sell their soul.md and hype for the madness.

Shoggoth.db : a self organizing database to experiment with agents without having to let them roam freely.

Posted a show hn earlier today that didn't got any traction : https://news.ycombinator.com/item?id=47738516

Happy Easter HN

This year for my fun Easter project, I decided to revisit my HN first project from 10 years ago (a chatbot based on Wikidata), but with the technology and vibe of the day.

So this is in the same vibe as Karpathy's LLM Wiki : https://gist.github.com/karpathy/442a6bf555914893e9891c11519... but with more limited permissions.

So let me introduce Shoggoth.db, a self building, self organising database.

Everything is done locally : single agent QWen3.5 35BA13 with limited network access to a single local wikix website, a a sqlite3 database, and initial prompt :

Which set his goal to explore the full wikipedia wiki, and build a dataset of mythological creatures and some facts about them.

The work is split in a sequence of short sessions, starting from scratch each time. So the fresh agent will discover the current state of the database and try its best to improve it during the work session.

Happy fun exploring safely.

The model does call tools successfully giving sensible parameters but it seems to not picking the right ones in the right order.

I'll try in a few days. It's great to be able to test it already a few hours after the release. It's the bleeding edge as I had to pull the last from main. And with all the supply chain issues happening everywhere, bleeding edge is always more risky from a security point of view.

There is always also the possibility to fine-tune the model later to make sure it can complete the custom task correctly. But the code for doing some Lora for gemma4 is probably not yet available. The 50% extra speed seems really tempting.

I just tried with llama.cpp RTX4090 (24GB) GGUF unsloth quant UD_Q4_K_XL You can probably run them all. G4 31B runs at ~5tok/s , G4 26B A4B runs at ~150 tok/s.

You can run Q3.5-35B-A3B at ~100 tok/s.

I tried G4 26B A4B as a drop-in replacement of Q3.5-35B-A3B for some custom agents and G4 doesn't respect the prompt rules at all. (I added <|think|> in the system prompt as described (but have not spend time checking if the reasoning was effectively on). I'll need to investigate further but it doesn't seem promising.

I also tried G4 26B A4B with images in the webui, and it works quite well.

I have not yet tried the smaller models with audio.

TLDR: It's easy : LLM outputs are untrusted. Agents by virtue of running untrusted inputs are malware. Handle them like the malware they are.

>>> "While this web site was obviously made by an LLM" So I am expecting to trust the LLM written security model https://jai.scs.stanford.edu/security.html

These guys are experts from a prestigious academic institution. Leading "Secure Computer Systems", whose logo is a 7 branch red star, which looks like a devil head, with white palm trees in the background. They are also chilling for some Blockchain research, and future digital currency initiative, taking founding from DARPA.

The website also points towards external social networks for reference to freely spread Fear Uncertainty Doubt.

So these guys are saying, go on run malware on your computer but do so with our casual sandbox at your own risk.

Remember until yesterday Anthropic aka Claude was officially a supply chain risk.

If you want to experiment with agents safely (you probably can't), I recommend building them from the ground up (to be clear I recommend you don't but if you must) by writing the tools the LLM is allowed to use, yourself, and by determining at each step whether or not you broke the security model.

Remember that everything which comes from a LLM is untrusted. You'll be tempted to vibe-code your tools. The LLMs will try to make you install some external dependencies, which you must decide if you trust them or not and review them.

Because everything produced by the LLM is untrusted, sharing the results is risky. A good starting point, is have the LLM, produce single page html page. Serve this static page from a webserver (on an external server to rely on Same Origin Policy to prevent the page from accessing your files and network (like github pages using a new handle if you can't afford a vps) ). This way you rely on your browser sandbox to keep you safe, and you are as safe as when visiting a malware-infested page on the internet.

If you are afraid of writing tools you can start by copy-pasting, and reading everything produced.

Once you write tools, you'll want to have them run autonomously in a runaway loop taking user feedback or agent feedback as input. But even if everything is contained, these run away loop can and will produce harmful content in your name.

Here is such vibe-coded experiment I did a few days ago. A simple 2d physics water molecules simulation for educational purposes. It is not physically accurate, and still have some bugs, and regressions between versions. Good enough to be harmful. https://news.ycombinator.com/item?id=47510746

He even attempts to improve on the paper by replacing the random rotation operation which is O(d^2), by a Subsampled Randomized Hadamard Transform which can be computed in O(d*log d).

Hopefully Johnson–Lindenstrauss lemma applies in the same way for SRHTransformed vectors as they do for randomly rotated vectors and the independence of the distribution laws of the coordinates remains and therefore the quantization of each coordinates independently is still theoretically sound.

A little vibecoded experiment I did today.

Using local only model with llama.cpp, using Qwen3.5-35B-A3B-UD-Q4_K_XL, copy pasting the full generated html page (no advance editor which do incremental file modification :) )

30 versions ~200k tokens one afternoon.

You can view all the files on https://github.com/unrealwill/watermolecules

You can also open the various files directly, some work, some are buggy : For example https://unrealwill.github.io/watermolecules/watermolecule20.... is OK.

https://unrealwill.github.io/watermolecules/watermolecule19.... is a funny bug.

I was just playing writing a 2d chemical simulator to simulate liquids at the molecular level.

It's not physics accurate at all. For it to be considered a liquid, it needs to have the molecule close enough that molecular force have an impact. I display the average distance to the nearest neighbor. At normal condition of temperature and pression, the average distance between water molecules should be roughly 3 angstroms.

Here the average will depend on the number of particles and the size of your browser window and zoom level.

The Van der Waals forces (the force between the molecules) have a range of 50 in this simulation, so this is something in between a liquid and a gas.

The rotations of the molecules are just for the visual effect.

I have also added auto-ionization of water. And chemical reactions.

Energy conservation and other conservation are not conserved. The simulation is more stable on the "high" level of quality.

Probably won't update further, but some agents may use your various remarks to create better simulations.

I had to check the precise definition for trigonometry while writing my comment, found it interesting so I added a reference.

As with many subject that we learn early in school, it's often interesting revisiting them as adult to perceive additional layer of depth by casting a new look.

With trigonometry we tend to associate it with circle. But fundamentally it's the study of tri-angles.

What is interesting is that the whole theory is "relative". I would reference the wikipedia page for angle but it may make me look like an LLM. The triangle doesn't have positions and orientation baked-in, what matters is the length of the sides and the angle between them.

The theory by definition becomes translation and rotation invariant. And from this symmetry emerge the concept of rotations.

What is also interesting about the concept of angle is that it is a scalar whereas the original objects like lines live in an higher dimension. To avoid losing information you therefore need multiple of these scalars to fully describe the scene.

But there is a degree of redundancy because the angles of a triangle sums to pi. And from this degree of freedom results multiple paths to do the computations. But with this liberty comes the risks of not making progress and going in circles. Also it's harder to see if two points coming from different paths are the same or not, and that's why you have "identities".

Often for doing the computation it's useful to break the symmetry, by picking a center, even though all points could be centers, (but you pick one and that has made all the difference).

Similar situation arise in Elliptic Curve Cryptography, where all points could have the same role, but you pick one as your generator. Also in physics the concept of gauge invariance.

I think it boils down to the alternate view of rotations as two successive reflections.

You can then use householder matrix to avoid trigonometry.

These geometric math tricks are sometimes useful for efficient computations.

For example you can improve Vector-Quantization Variational AutoEncoder (VQ-VAE) using a rotation trick, and compute it efficiently without trigonometry using Householder matrix to find the optimal rotation which map one vector to the other. See section 4.2 of [1]

The question why would someone avoid trigonometry instead of looking toward it is another one. Trigonometry [2] is related to the study of the triangles and connect it naturally to the notion of rotation.

Rotations [3] are a very rich concept related to exponentiation (Multiplication is repeated addition, Exponentiation is repeated multiplication).

As doing things repeatedly tend to diverge, rotations are self stabilizing, which makes them good candidates as building blocks for the universe [4].

Because those operations are non commutative, tremendous complexity emerge just from the order in which the simple operations are repeated, yet it's stable by construction [5][6]

[0]https://en.wikipedia.org/wiki/Householder_transformation

[1]https://arxiv.org/abs/2410.06424

[2]https://en.wikipedia.org/wiki/Trigonometry

[3]https://en.wikipedia.org/wiki/Matrix_exponential

[4]https://en.wikipedia.org/wiki/Exponential_map_(Lie_theory)

[5]https://en.wikipedia.org/wiki/Geometric_algebra

[6]https://en.wikipedia.org/wiki/Clifford_algebra

Interesting work. It's a nice introduction to usage of holography.

We can etch the inside of a photosensitive material by focusing a laser at a specific point, and moving this point of focus. That's what is done in [3] [4].

But here instead of doing this sequentially they print all points simultaneously using holography.

Here they use holography to light volumetrically some photosensitive resin, in a similar fashion as it used to be done for volumetric display (In [2] you can find a figure of using an agarose gel tank as a display for volumetric hologram).

They just put a new "spin" on it, by spinning mirror around the resin tank, to project from all direction, to be able to the back of objects. The technique called "Digital Incoherent Synthesis of Holographic Light Fields" paints the resin bath with a 3D-paint brush, sequentially from multiple direction. It's called incoherent because each angle is treated independently from the other and it's light doesn't need to be interfered (in the wave sense).

The natural extension of using a conical mirror instead of spinning the mirror would need to also consider the interference of light of nearby angles making the inverse problem computation harder, and need to have higher resolution, but would avoid moving objects.

Here the holography is a fancy way of focusing the light where we want the resin to cure. It needs to have a resin with optical properties which don't change once cured or then the light behind the cured resin won't be focused where it should, even though it should cure everywhere simultaneously. Unfocused light is still absorbed by the resin which contributes to the curing, but photosensitive resin are non linear meaning nothing happens until you cross a threshold.

To do this holography, they use Digital Micromirror Devices (DMD) : a chip which has an array of micro mirror pixels. [0]

Although these mirrors are on-off only, some technique from the 1970s, allows you to control the shape of the light-field, in amplitude, phase and polarization [1].

DMD is a chip located inside the widely available technology available in Digital Light Projector. They are used as display and that's how you control the mirrors. You use them as a screen over the HDMI interface to display the right pattern. Then you just have to bounce some laser on it to have a "structured light" beam, from which you use a few lenses and hole (in a 4f arragement) to extract the "mode" you are interested whose light will refocus itself at the right 3d points.

The limits of this technique is due to the resolution of the DMD (as explained in [2]), where the smaller the pixel size the better. But here this limit is mitigated by integrating over time and angle, because what matters is resin exposition time.

[0] "Structuring Light with Digital Micromirror Devices (Photonics West 2021)" https://www.youtube.com/watch?v=vurtdU0FRm4

[1] Binary amplitude holograms for shaping complex light fields with digital micromirror devices https://www.institut-langevin.espci.fr/biblio/2025/1/18/2280...

[2] Holographic video display using digital micromirrors (Invited Paper) [the raven has the key]

[3] "how to put 3D images into glass or crystal objects 3d crystal Inside carving" https://www.youtube.com/watch?v=dkK6c45U6EU

[4] "What is Sub-surface Laser Engraving or a 'Bubblegram'? Technology Explained" https://www.youtube.com/watch?v=sOrby692Uag

It's even worse than that.

The positives outcomes are structurally being closed. The race to the bottom means that you can't even profit from it.

Even if you release something that have plenty of positive aspects, it can and is immediately corrupted and turned against you.

At the same time you have created desperate people/companies and given them huge capabilities for very low cost and the necessity to stir things up.

So for every good door that someone open, it pushes ten other companies/people to either open random potentially bad doors or die.

Regulating is also out of the question because otherwise either people who don't respect regulations get ahead or the regulators win and we are under their control.

If you still see some positive door, I don't think sharing them would lead to good outcomes. But at the same time the bad doors are being shared and therefore enjoy network effects. There is some silent threshold which probably has already been crossed, which drastically change the sign of the expected return of the technology.

I like the game up until 7 or 8 notes, but it keeps adding note.

I couldn't find a setting to freeze the difficulty where it's comfortable and where the melody can still be construed to make sense.

When adding more notes, it breaks the flow and turn a training for pitch practicing into a memory game for rain man, even more so when we make a mistake and must redo the melody partially.

Have you checked his physical keyboard ?

My laptop is getting old, and some keys need to be pressed with more insistence and more accurately for them to register properly. It also breaks the flow, and muscle memory for things like passwords. It also lead to letter inversions, because the tonic accent need to be put on letter which need to be pressed more, rather than on the first letter of the word. It's driving me crazy but unfortunately computer are too expensive for now (and it's probably only getting worse).

Or use lasers : https://www.youtube.com/watch?v=DCQ2CbfGs6g&t=440s

Please forgive my outside the box use of word.

I used it at the time as a reference to the "PNG aCropalypse" ( https://news.ycombinator.com/item?id=35208721 where I originally shared it in a comment).

The algorithm does also work if you remove the outer areas of the photo.

And that's just in the non-adversarial simple case.

If you don't know the provenance of images you are putting black box on (for example because of a rogue employee intentionally wanting to leak them, or if the image sensor of your target had been compromised to leak some info by another team), your redaction can be rendered ineffective, as some images can be made uncroppable by construction .

(Self-plug : https://github.com/unrealwill/uncroppable )

And also be aware that compression is hiding everywhere : https://en.wikipedia.org/wiki/Compressed_sensing

If it's not done properly, and you happen at any point in the chain to put black blocks on a compressed image (and PDF do compress internal images), you are leaking some bits of information in the shadow casted by the compression algorithm : (Self-plug : https://github.com/unrealwill/jpguncrop )

The problem of computers is the problem of time : How to obtain a consistent causal chain !

The classical naive way of obtaining a consistent causal chain, is to put the links one after the other following the order defined by the simulation time.

The funnier question is : can it be done another way ? With the advance of generative AI, and things like diffusion model it's proven that it's possible theoretically (universal distribution approximation). It's not so much simulating a timeline, but more sampling the whole timeline while enforcing its physics-law self-consistency from both directions of the causal graph.

In toy models like game of life, we can even have recursivity of simulation : https://news.ycombinator.com/item?id=33978978 unlike section 7.3 of this paper where the computers of the lower simulations are started in ordered-time

In other toy model you can diffusion-model learn and map the chaotic distribution of all possible three-body problem trajectories.

Although sampling can be simulated, the efficient way of doing it necessitate to explore all the possible universes simultaneously like in QM (which we can do by only exploring a finite number of them while bounding the neighbor universe region according to the question we are trying to answer using the Lipschitz continuity property).

Sampling allows you to bound maximal computational usage and be sure to reach your end-time target, but at the risk of not being perfectly physically consistent. Whereas simulating present the risk of the lower simulations siphoning the computational resources and preventing the simulation time to reach its end-time target, but what you could compute is guaranteed consistent.

Sampled bottled universe are ideal for answering question like how many years must a universe have before life can emerge, while simulated bottled universe are like a box of chocolate, you never know what you are going to get.

The question being can you tell which bottle you are currently in, and which bottle would you rather get.

What they need is not so much memory but memory bandwidth.

For training, their models have a certain number of memory needed to store the parameters, and this memory is touched for every example of every iteration. Big models have 10^12 (>1T )parameters, and with typical values of 10^3 examples per batch, and 10^6 number of iteration. They need ~10^21 memory accesses per run. And they want to do multiple runs.

DDR5 RAM bandwidth is 100G/s = 10^11, Graphics RAM (HBM) is 1T/s = 10^12. By buying the wafer they get to choose which types of memory they get.

10^21 / 10^12 = 10^9s = 30 years of memory access (just to update the model weights), you need to also add a factor 10^1-10^3 to account for the memory access needed for the model computation)

But the good news is that it parallelize extremely well. If you parallelize you 1T parameters, 10^3 times, your run time is brought down to 10^6 s = 12 days. But you need 10^3 *10^12 = 10^15 Bytes of RAM by run for weight update and 10^18 for computation (your 120 billions gigabytes is 10^20, so not so far off).

Are all these memory access technically required : No if you use other algorithms, but more compute and memory is better if money is not a problem.

Is it strategically good to deprive your concurrents from access to memory : Very short-sighted yes.

It's a textbook cornering of the computing market to prevent the emergence of local models, because customers won't be able to buy the minimal RAM necessary to run the models locally even just the inferencing part (not the training). Basically a war on people where little Timmy won't be able to get a RAM stick to play computer games at Xmas.

In the video, in the continuous version the game never end and highlight the "loser" strategy.

When you are behind the optimal play is to make a gamble, which most likely will make you even worse. From the naive winning side it seems the loser is just doing a stupid strategy of not following the optimal dichotomy strategy, and therefore that's why they are losing. But in fact they are a "player" doing not only their best, but the best that can be done.

The infinite sum of ever smaller probabilities like in Zeno's paradox, converge towards a finite value. The inevitable is a large fraction of the time, you are playing catch-up and will never escape.

You are losing, playing optimally, but slowly realising the probabilities that you are a loser as evidence by the score which will most likely go down even more next round. Most likely the entire future is an endless sequence of more and more desperate looking losing bets, just hoping to strike it once that will most likely never comes.

In economics such things are called "traps", for example the poverty trap exhibits similar mechanics. Where even though you display incredible ingenuity by playing the optimal game strategy, most of the time you will never escape, and you will need to take even more desperate measures in the future. That's separating the wheat from the chaff from the chaff's perspective or how you make good villains : because like Bane in Batman there are some times (the probability is slim but finite) where the gamble pays off once and you escape the hell hole you were born in and become legend.

If you don't play this optimal strategy you will lose slower but even more surely. The optimal strategy is to bet just enough to go from your current situation to the winning side. It's also important not to overshoot : this is not always taking moonshots, but betting just enough to escape the hole, because once out, the probabilities plays in your favor.

The theoretical maximum rate for 1B particle advection (Just doing p[] += v[]dt), is 1000GB/s / 24GB = 42 iteration per second. If you only have 100M you can have 10 times more iteration.

But that's without any rendering, and non interacting particles which are extremely boring unless you like fireworks. (You can add a term like v[] += gdt for free.) And you don't need to store colors for your particles if you can compute the colors from the particle number with a function.

Rasterizing is slower, because each pixel of the image might get touched by multiple particles (which mean concurrent accesses in the GPU to the same memory address which they don't like).

Obtaining the screen coordinates is just a matrix multiply, but rendering the particles in the correct depth order requires multiple pass, atomic operations, or z-sorting. Alternatively you can slice your point clouds, by mixing them up with a peak-shaped weight function around the desired depth value, and use an order independent reduction like sum, but memory accesses are still concurrent.

For the rasterizing, you can also use the space partitioning indices of the particle to render to a part of the screen independently without concurrent access problems. That's called "tile rendering". Each tile render the subset of particles which may fall in it. (There are plenty of literature in the Gaussian Splatting community).

TLDR : 1B particles ~ 3s per iterations

For examples like particle simulations, on a single node with a 4090 GPU everything running on GPU without memory transfer to the CPU:

-The main bottleneck is memory usage : available 24GB, Storing the particles 3 position coordinates, + 3 velocity coordinates, 4 bytes by number (float32) = Max 1B particles

-Then GPU memory bandwidth : if everything is on the GPU you get between 1000GB/s of global memory access and 10000GB/s when shared memory caches are hit. The number of memory access is roughly proportional to the number of effective collisions between your particles which is proportional to the number of particles so around 12-30 times ( see optimal sphere packing number of neighbors in 3d, and multiply by your overlap factor). All in all for 1B particles, you can collision them all and move them in 1 to 10s.

If you have to transfer things to the CPU, you become limited by the PCI-express 4.0 bandwidth of 16GB/s. So you can at most move 1B particles to and from the GPU, 0.7 times per second.

Then if you want to store the particle on disk, instead of RAM because your system is bigger, then you can either use a M2 ssd (but you will burn them quickly) which has a theoretical bandwidth of 20GB/s so not a bottleneck, or use a network storage over 100Gb/s (= 12.5GB/s) ethernet, via two interfaces to your parameter server which can be as big as you can afford.

So to summarize so far : 1B particles takes 1 to 10s per iteration per GPU. If you want to do smarter integration schemes like Rk4, you divide by 6. If you need 64 bits precisions you divide by 2. If you only need 16bits precisions you can multiply by 2.

The number of particle you need : Volume of the box / h^3 with h the diameter of the particle = finest details you want to be able to resolve.

If you use an adaptive scheme most of your particles are close to the surface of objects so O( surface of objects / h^2 ) with h=average resolution of the surface of the mesh. But adaptive scheme is 10 times slower.

The precision of the approximation can be bounded by Taylor formula. SPH is typically order 2, but has issues with boundaries, so to represent a sharp boundary the h must be small.

If you want higher order and sharp boundaries, you can do Finite Element Method, instead. But you'll need to tessellate the space with things like Delaunay/Voronoi, and update them as they move.

There are two types of stops : soft stops, and hard stops.

- Soft stops is when the dynamic graph computation overhead is too much, which mean you can still calculate, but if you were to write the function manually or with a better framework, you could be 10x faster.

Typical example involve manually unrolling a loop. Or doing kernel fusion. Other typical example is when you have lots of small objects or need to do loops in python because it doesn't vectorize well. Or using the sparsity efficiently by ignoring the zeros.

- Hard stop is when computing the function become impossible, because the memory needed to do the computation in a non optimal way explode. Some times you can get away with just writing customized kernels.

The typical example where you can get away with it are custom attention layers.

Typical example where you can't get away are physics simulations. Like for example the force is the gradient of energy, but you have n^2 interactions between the particles, so if you use anything more than 0 memory preserved during the forward pass per interaction, your memory consumption explode. And typically with things like Lagrangian or Hamiltonian neural networks where you look the discover dynamics of an energy conserving system, you need to be able differentiate at least three times in a row.

There are also energy expanding stops, where you need to find work-around to make it work like if you want to have your parameters changing shape during the optimization process like learning point clouds of growing size, and they spread you thin so they won't be standardized.

> a lot of network lock-in that an upstart would have to overcome by being way better at something PyTorch can never be good at

Higher level code migration to the newer framework, is going to 0. You ask your favorite agent (or intern) to port and check that the migration is exact. We already see this in the multitude of deep-learning frameworks.

The day one optimization trick that PyTorch can't do but another framework can, which reduce your training cost 10x and PyTorch is going the way of the dodo.

The day one architecture which can't be implemented in PyTorch get superior performance, and it's bye bye python.

We see this with architectures which require real-time rendering like Gaussian Splatting (Instant Nerf), or the caching strategies for LLM sequence generation.

Pytorch's has 3 main selling point :

- Abstracting away the GPU (or device) specific code, which is due to nvidia's mess : custom optimized kernels, which you are forced to adapt to if you don't want to write custom kernels.

If you don't mind writing optimized kernels, because the machine write them. Or if you don't need Cuda because you can't use Nvidia hardware because for example you are in China. Or if you use custom silicon, like Grok and need your own kernels anyway.

- Automatic differentiation. It's one of its weak point, because they went for easy instead of optimal. They shut themselves off some architectures. Some language like Julia because of the dynamic low-level compilation can do things Pytorch won't even dream about, (but Julia has its own problems mainly related to memory allocations). Here with the pytorch's introduction of the "scan function"[2] we have made our way full circle to Theano, Tensorflow's/Keras ancestor, which is usually the pain point of the bad automatic differentiating strategy chosen by Pytorch.

The optimal solution like all physics Phds which wrote simulations know, is writing custom adjoint code with 'Source Code Transformation' or symbolically : it's not hard but very tedious so it's now a great fit for your LLM (or intern or Phd candidate running 'student gradient descent') if you prove or check your gradient calculation is ok.

- Cluster Orchestration and serialization : a model can be shared with less security risks than arbitrary source code, because you only share weights. A model can be splitted between machines dynamically. But this is also a big weakness because your code rust as you become dependent of versioning, you are locked with the specific version number your model was trained on.

[2] "https://docs.pytorch.org/xla/master/features/scan.html

The choice of the dynamic computation graph [1] of PyTorch made it easier to debug and implement, leading to higher adoption, even though running speed was initially slower (and therefore training cost higher).

Other decisions follow from this one.

Tensorflow started with static and had to move to dynamic at version 2.0, which broke everything. Fragmentation between tensorflow 1, tensorflow 2, keras, jax.

Pytorch's compilation of this computation graph erased the remaining edge of Tensorflow.

Is the battle over ? From a purely computational point, Pytorch solution is very far from optimal and billions of dollars of electricity and GPUs are burned every year, but major players are happy with circular deals to entrench their positions. So at the pace of current AI code development, probably one or two years before Pytorch is old history.

[1] https://www.geeksforgeeks.org/deep-learning/dynamic-vs-stati...

>computing the exact gradient using calculus

First of all, gradient computation with back-prop (aka reverse-mode automatic differentiation) is exact to numerical precision (except for edge-cases that are not relevant here) so it's not about the way of computing the gradient.

What Andrej is trying to tell is that when you create a model, you have freedom of design in the shape of the loss function. And that in this design what matters for learning is not so much the value of the loss function, but its slopes, and curvature (peaks and valleys).

The problematic case being flat valleys, surrounded by straight cliffs, (picture the grand canyon).

Advanced optimizers in deep learning like "Adam", are still first-order, with diagonal approximation of the curvature, which mean the optimizer in addition to the gradient it has an estimate of the scale sensitivity of each parameter independently. So the cheap thing it can reasonably do is modulate the gradient with this scale.

The length of the gradient vector, being often problematic, what optimizers would usually do was something called "line-search", which is determine the optimal step-size along this direction. But the cost of doing that is usually between 10-100 evaluation of the cost function which is often not worth the effort in the noisy stochastic context, compared to just taking a smaller step multiple times.

Higher-order optimizers necessitate that the loss function is twice differentiable, so non-linearities like relu, which are cheap to calculate can't be used.

Lower-order global optimizers don't even necessitate the gradient, which is useful when the energy-function landscape has lots of local minima, (picture an egg-box).