t-vi

It's a yet to be published feature of SERA apparently. I would expect more CliffsNotes than jump a Cliff, I guess.

Not sure about the link, the official landscape URL seems to be this (same data probably):

https://landscape.pytorch.org/

We do try to check the pulse of projects there with a checklist, but I guess it started in a more uncoordinated way.

If you file an issue here, I think it would work to update things:

https://github.com/pytorch-fdn/ecosystem

They have some pretty cool people, though, no reason not to think they'll catch up soon enough.

- I don't think it hurts to learn PyTorch (and having learned JAX is good, too). I don't know if JAX + triton is as impossible as you make it out, but it seems that PyTorch integration is quite good for many things. - For pallas, triton and CUDA/C++, you probably want to know a bit about how GPU works. There is the GPU-Mode discord / lectures / ressources if you are looking for material https://github.com/gpu-mode/ . - In my experience how well Triton works varies depending on what you want to do (depending on the how well the programming model fits the task). If it does, it is quite nice to get something reasonably fast reasonably fast. PyTorch (in the inductor torch.compile backend) has made many things work well, so you could check that out if you run out of examples elsewhere).

Note that the NVIDIA container uses CUDA+cuBLAS 13.0.2 which cites "Improved performance on NVIDIA DGX Spark for FP16/BF16 and FP8 GEMMs", which seems to be your use-case. In general, I would suspect that it mostly comes to versions of the libs.

Interestingly, there is a cuBLAS 13.1 whl on PyPI, not sure what that does.

It seems to me that in 2016 people did (have to) play a lot more tricks with the backpropagation than today. Back then it was common to meddle with gradients in between the gradient propagation.

For example, Alex Graves's (great! with attention) 2013 paper "Sequence Generation with Recurrent Neural Networks" has this line:

One difficulty when training LSTM with the full gradient is that the derivatives sometimes become excessively large, leading to numerical problems. To prevent this, all the experiments in this paper clipped the derivative of the loss with respect to the network inputs to the LSTM layers (before the sigmoid and tanh functions are applied) to lie within a predefined range.

with this footnote:

In fact this technique was used in all my previous papers on LSTM, and in my publicly available LSTM code, but I forgot to mention it anywhere—mea culpa.

That said, backpropagation seems important enough to me that I once did a specialized videocourse just about PyTorch (1.x) autograd.

Note that PyTorch's kernels are somewhat generic in shape. It has always been relatively easy to get speedups by specializing the shape, e.g. Apache TVM had that (back before it was "Apache" even).

If you like JIT wrappers and Python interpreters:

In Thunder[1], a PyTorch to Python JIT compiler for optimizing DL models, we are maintaining a bytecode interpreter covering 3.10-3.12 (and 3.13 soon) for our jit. That allows to run Python code while re-directing arbitrary function calls and operations but is quite a bit slower than CPython.

While the bytecode changes (and sometimes it is a back-and-forth for example in the call handling), it seems totally good once you embrace that there will be differences between Python versions.

What has been a large change is the new zero cost (in the happy path) exception handling, but I can totally why Python did that change to that from setting up try-block frames.

I will say that I was happy not to support Python <= 3.9 as changes were a lot more involved there (the bytecode format itself etc.).

Of course, working on this has also means knowing otherwise useless Python trivia afterwards. One of my favorites is how this works:

  l = [1, 2, 3]
  l[-1] += l.pop()
  print(l)

1. https://github.com/Lightning-AI/lightning-thunder/

When I did a similar thing (but with less LLM) I liked https://github.com/coqui-ai/TTS but back then I needed to cut out the conversion step from tensor to a list of numbers to make it work really nicely.

The subtraction is because "is an example of constructing the “Inner Product” distance" per the text above it. That ymmone might not be one could be because they only need that up to a constant and so don't care, but it's probably not ideal to name the thing containing 0s ymmone.

> Is avoiding CF potentially just a matter of sheer scale ?

My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.

After the first epoch, the average time since the present data item was last used for during training is small at the beginning of an epoch grows during the epoch. I'd expect that to positively relate to loss on the present iteration.

My former neighbors run https://justanotherfoundry.com/ and I like their work and bought some.

To be honest, I never get what people want with all that business and wonder if it is because the abstraction ("ordered derivatives") implied is not ideal.

If we follow the ordinary chain rule (for a single coordinate if you want) through the edges of the computational (DAG) graph, we get the right thing in each step.

The only other rule you need is that "if you use one variable several times in a calculation (i.e. several edges from(fw)/to(bw) the same node), you need to add the gradients computed for each", but IMHO that is pretty basic and intuitive, too. (So if you plug in z for both x and y into f(x, y), you have d/dz f(z, z) = f_x(z, z) + f_y(z, z), where the subscript indicates partial derivative.)

To me this seems both mathematically simpler than mixing the two into a "more than chain rule" thing and closer to what is actually going on algorithmically in a given implementation (the one I'm most familiar with is probably PyTorch's).

The PEP says that a naïve implementation would cause a 4% performance hit and have a list of various ways of speeding up cases where immortality makes sense / is guaranteed that they say can bring this to parity. Maybe they implemented part of the list.

This special casing just had not been implemented yet. But as it is an interesting optimization, more so with multi-interpreter or no-GIL Python, the developers will actually introduce immortal objects in Python 3.12 to avoid counting references on some objects (PEP 683 has been accepted):

https://peps.python.org/pep-0683/

Not saying that the vector space bit isn't neat, but it's called functional analysis because you can take limits of various forms and define (semi-) continuity, have completions of spaces, and all that has nice properties. So to me, a crucial thing is that these vector spaces are indeed topological.

It knows nothing of the modern stuff (because MacKay died too early), but skipping the first parts of David MacKay: Information Theory, Inference, and Learning Algorithms you get a very accessible course in (200x) Bayesian Inference that should cover most of what you need for diving into PPL applications.

http://www.inference.org.uk/mackay/itila/book.html

The logic error seems to be that you have the second for loop where you only want to destructure the splitted line into a tuple. Now writing that decomposition as a for loop over a one-element list (as in the corrected code) works, but seems "hmhm". In other words, the misunderstanding here is that you want a chain of generators, not a "nested for loop"-generator.

A more proper solution could be to chain the generator expressions: with

    lines = ["a,b", "c,d"]

you could do

    ((a,b) for a, b in (l.split(',') for l in lines))

I use Debian's system Python 3.10 install for most of my stuff and it works really well for me. Some things I install via pip but the key libraries (e.g. PyTorch) from source.

There is A. Karpathy's recipe for training NNs but it is not a walkthrough with an example:

https://karpathy.github.io/2019/04/25/recipe/

but the general idea of "get something that can overfit first" is probably pretty good.

In my experience getting the data right is probably the most underappreciated thing. Karpathy has data as step one, but in my experience, also data representation and sampling strategy does quite the miracle.

In Part II of our book we do an end-to-end project including e.g. a moment where nothing works until we crop around "regions of interest" to balance the per-pixel classes in the training data for the UNet. This has been something I have pasted into the PyTorch forums every now and then, too.

The adapter and LoRa have a drastically fewer parameters, so one might expect that forward + backward is roughly 2x the cost of forward.

Then (as far as I know), in contrast to generation, training is done on the entire output of the transformer (so all tokens of the full input) rather than serially token-by-token (in the RNN days, this was called teacher-forcing), so that may give you a significant boost in the tokens per second rate over generation.

Out of curiosity, how do you handle things where the output shape is input dependent (as opposed to only dependent on input shapes)? This is from `torch.sum(tensor, dim)` where dim might be nonconstant to `torch.nonzero(x)` and of course advanced indexing.

On the other hand, it would fix cases where there are somewhat rare hangs that are hard to reproduce, for example in multithreaded applications embedding Python. A while ago, I hit a deadlock in LibreOffice's Python extension system. I managed to fix it by changing the GIL to the more modern Python API for that, but I never managed to produce a reasonably small test case, so I didn't submit it.

Both:

A long while ago, I wrote a little tutorial[0] on quantizing a speech commands network to the Raspberry. I used that to control lights directly and also for wake word detection.

More recently, I found that I can just use more classic VAD because my uses typically don't suffer if I turn on/off the microphone. My main goal is to not get out the mobile phone for information. That reduces the processing when I turn on the radio...

Not high-end as your solution, but nice enough for my purposes.

[0]. https://devblog.pytorchlightning.ai/applying-quantization-to...

Personally, I plugged a Jabra conference speaker to a Raspberry and if it hears something interesting, it sends to my local GPU computer for decoding (with whisper) + answer-getting + response sent back to the Raspberry as audio (with a model from coqui-ai/TTS but using more plain PyTorch). Works really nicely for having very local weather, calendar, ...

Not to forget CPython's own faster-cpython project which aims at JIT compiling. [0]

Also JAX[1], PyTorch[2] come with JIT compilation specifically aimed at GPU kernels "fusing" multiple higher-level operation

And NumPy/Scipy (also) uses Pythran[3], an AOT compiler not too unsimilar to Numba.

[0] https://github.com/faster-cpython/cpython [1] https://pytorch.org/docs/stable/generated/torch.compile.html [2] https://jax.readthedocs.io/ [3] https://pythran.readthedocs.io/

I think some useful classification criteria would be - does it replace running code in Python (either own interpreter or compiler), vs does it speed up certain bits, - does it aim to faithfully implement Python or does it intentionally diverge in the semantics, - does it provide low-level semantics (where numba, pythran shine) or higher-level (e.g. what PyTorch, JAX do) - target architectures (CPU, GPU offloading, ...)

In fact, counting federal, state and local governments, Germany has some 6 million government employees (~4.8m directly and ~1.2m in public agencies). Also, there might be things that are government provided in one country and private in another (postal service, telcom and utilities, railway, schools, childcare...), which you would need to account for as well.

Do they need to learn rules though, or could they memorize enough to learn the probabilities of the most likely continuation? To my mind the blurry-jpeg-metaphor[1] is very much spot-on. While it is speculation, it would seem to me personally that LLMs de-facto doing a (lossy) memorization and neural networks in general being able to do well in "local consistency" seems a way to think about them that is consistent with most of the model behaviour we observe.

1. https://www.newyorker.com/tech/annals-of-technology/chatgpt-...