118 karma · joined November 22, 2019
While I'm not going to disagree that it matters, to my mind the main thing about early (0.x, 1.x) PyTorch's success was a clear vision (most visible to me from Soumith and Adam) that put the targeted users first (as in your productivity comment), awesome execution on it, and a creating a community where I, as an outside contributor with modes skill and no AI track record, and likely many others, felt welcome.
Now, the many great people employed by Meta, but also other players like MS, NVidia, AMD, Intel, ... to work on PyTorch enabled all the things that make PyTorch 2.0 nicer and more broadly applicable than 1.0. (And to my mind it's quite a non-trivial accomplishment to enable other corporate players to join the party.)
That said I think the results are often useful and sometimes fascinating. We should not fool ourselves about the learning that these large neural nets do, though.
It's a bit tricky to go really deep with tests when you don't have a reference, but I like to see that at least basic logic errors and typos in attribute names are caught.
Many of the larger libraries in the PyTorch ecosystem have some sort of testing. The things that are mostly "here is a model" often do not.
If, on the other hand, you take Python's core strengths like the simplicity, to my mind Julia does not necessarily match these. I have seen people being able to extend Python programs without any Python training or tutorials or even programming affinity just by imitating the code next to where they want the program to do something. I have a hard time imagining that that would be quite as easy with Julia.
Similarly, if you take the ideas from the Zen of Python, you might find that Python (code) fares better on them than Julia.
Now, any of the features that contribute to Julia being more complex in the above sense has a good reason for existing, but there is is value in Python's simplicity that can be easy to overlook.
Of course, one might wonder whether Python started to depart from that - with := and structural pattern matching being examples of things that are valuable features but at the same time adding quite a bit of complexity while being much more controversial than, say, list comprehensions.
Note that this is not about being able to write programs that are easy to understand but about any given code being easy to understand (at the "what does this line of code do" level).
It may also be instructive to compare the (English) Wikipedia entries of both, I think they are to some extend reflections of the different philosophies behind the two languages.
import torch
x = torch.randn(4000, 4000, device='cuda:0')
y = torch.randn(4000, 4000, device='cuda:0')
import time
torch.cuda.synchronize()
t0 = time.perf_counter()
z = torch.zeros(4000, 4000, device='cuda:0')
for j in range(4000):
z += (x[:, j] * z[j, :]).relu()
torch.cuda.synchronize()
t1 = time.perf_counter()
print(t1-t0)
Of course, the backward will be more work because you need to recompute which bits were 0.
I can appreciate that Python is slow and that Julia has very nice properties dissolving the boundaries of operators that are awefully hard in PyTorch & Co, but this is one of these examples where the story seems rather flawed (you don't need the for loop over 64 billion elements, just over the 4000 j-coordinates, and the Python overhead here isn't too bad, either). As far as I can tell, the large overhead here is from materializing the intermediate results in global memory. Personally, I would hope that the fusers might be able to fuse in the not to far future, so the entire thing can run on the GPU in one kernel, but until then, if Julia can do that, it'll have a huge advantage here.
Shameless plug: In my courses about efficient PyTorch, I always give the rule of thumb "avoid operations on individual elements, but things operating on hundreds+ elements should be fine". And the reason isn't as much just "Python is slow" but the overhead of the tensor objects.
Example: I bought a 16GB Radeon VII for €550 (including 19% VAT). That would be a decent value proposition for Deep Learning even today. However, the card appears to trade second hand(!) for €1300+ on e-bay, because you apparently can run 90-100 MH/sec or so (this is an estimate I found on the web, I don't engage in this type of thing). As a result, you would not be able to get consumer Vega with these performance characteristics for a good price even if AMD produced them.
The news here is that the PyTorch team and AMD are confident enough about the quality that they're putting it on the front page. This has been a long way in the making, and finally achieving official support is a great step for the team working on it.
Just as there are many different applications of deep learning, "Production" is quite heterogeneous (you put something behind a server, or integrate it in a large codebase written in $foo, or put it on some embedded device or phone, want to deploy on one or several machines with or without GPUs or other HW accelleration,...). I would not expect a single tool or method to be universal. This means that I would not expect either to be uniformly better than the other. But it also means that a tool has an edge if it allows deployment in a wide range of ways. In my view, it is nice that PyTorch has achieved a high level of consistency between the Python interface and other ways of using / deploying PyTorch. Also, PyTorch does have some vision of interop (eg with ONNX).
The other part is that when looking at a complete lifecycle, with detecting / analysing potential update needs, updating models and deploying the updates, it would seem beneficial to be able to seemlessly go back and forth. In my experience, PyTorch does a good job here (although I might also have ideas for further improvements), in that you can take, say, a JIT exported module and then look at it and it has much of the structure of how it was originally written. Taking a, say, tflite exported model and looking at what it does seemed to involve looking at low-level things much more.
> Jeremy’s talk offers the kind of exchange of ideas that makes an intellectual community vibrant and healthy.
Code of conduct enforcement never is nice or fun on any side of it. Here it has been terrible, and for all Jeremy has suffered, there is little reason to doubt that the people on the other side of the process were well-intentioned but apparently not well-equipped to do this better. The author of the linked post appears to have some regrets over how things went and offers a sincere apology. It might be a reasonable conclusion that the people in charge of this process might not be skilled at CoC enforcement. The need of training people for this might be a general item to learn here, too.
Of course, you can push for harsher judgement of those at fault, I'm afraid it would do little to make the world better.
In addition to the distress the process has caused to Jeremy (and probably, to a far lesser extend other people involved), the outcome also doesn't do the perceived problem that started it any good.
There are two things that strike me as as making Jeremy's talk unusual: - There appears to be a substantial discrepancy between the abstract and the actual talk. In the talk it would seem that roughly the first and last quarter are about primarily nbdev and fastdoc (in alignment with the abstract) and the center half is a rebuttal. - It seems uncommon to offer point-by-point rebuttals of something as a keynote talk. Maybe this is for a good reason.
Now, as someone who has organized events with speakers, I must admit I would not appreciate if someone had been invited to give a talk on one thing and then takes half the time to talk about how someone else is wrong (with a subtext of "I am right") instead. I have a certain desire for reliability when I'm organizing things and I would feel let down someone did this to me.
The other part that made me feel the talk is unfortunate relates directly to the complaint as described in the NumFocus CoC event report. I would personally not consider giving a point-by-point rebuttal of someone else's presentation as a talk. I must admit I could not really appreciate the humorous aspect of the talk - maybe it was too shallow for my tastes or I am just too stuck up - and the rebuttal segment struck me as quite uselessly negative. (If humour lives off some form of discrepancy, it might not be hard to see how "I like notebooks" at a Jupytercon is much less witty than "I don't like notebooks" at a Jupytercon.)
My concern with having such talks - regardless of that Joel does not mind - is that I would worry about sending a message of "give a talk, have your points ripped into pieces 2 years later" to speakers who might be less happy to have that done to them. This is the aspect where I would actually go so far to say that it might not been the best judgement to give this talk as it is presented. I think the same content as the middle part could have easily been framed in the positive "best practices for working with Jupyter notebooks" sense and it would have been a much better, friendlier and more useful talk.
I respect Jeremy and his work and how much he has helped people achieve things, and I am saddened by this entire affair. But to my mind, the talk actually was not up to his usual standards, neither does it do Jupytercon justice nor the good work that Jeremy showcasing nor the insights he is presenting.
So, using the CoC and making this a process of "is Jeremy at fault?" has not only has caused considerable harm all around, but has also done a disservice to a discussion of what type of tone we want to set at our events.
Note that this isn't about what people should or should not be allowed to say, but I would suggest that not everything covered by free speech actually is worth having at a conference.
So AMD got a lot of heat for not supporting Navi (RDNA) with ROCm, but it seems that they are weeding out the things keeping people from running it ( https://github.com/ROCmSoftwarePlatform/pytorch/issues/718 and the links in that look like gfx10 is almost there for rocBlas and MIOpen). We'll see what ROCm 3.9 will bring and what the state of big navi is.
But I must admit I don't see the 2GB model as a better value than the $100 4GB one, especially with shared CPU/GPU.
But Paul's situation is multithreaded and his analysis has numbers that seem to indicate that something is up with the GIL. We know is a limitation in multithreaded PyTorch due to any Tensor creation at the Python level needing the GIL and these models typically creating quite a few of them.
It's always easier to know how the performance impact of something is when you have an experiment removing the bits. Maybe the using the JIT or moving things to C++ gives us that, I look forward to seeing a sequel.
The advantage of involving something like TensorRT or TVM is that they'll apply holistic optimizations - they may eliminate writing to memory and reading back (which would not show as underutilized GPU, but can be a big win, see e.g. the LSTM speedup with the PyTorch JIT fuser). The current disadvantage of TVM is that TVM currently is a bit of an all-or-nothing affair, so you can't give it a JITed model and say "optimize the bits that you can do well". TensorRT with TRTorch is a bit ahead there.
Of course, PyTorch itself is getting better too, with the new profiling executor and new fusers for the PyTorch JIT, so we might hope that you can have good perf for more workloads with just PyTorch.
When you have multithreaded setups, this typically is more significant than the Python overhead itself (which comes in at 10% for the PyTorch C++ extension LLTM example, but would be less for convnets).
At least for the PyTorch bits of it, using the PyTorch JIT works well. When you run PyTorch code through Python, the intermediate results will be created as Python objects (with GIL and all) while when you run it in TorchScript, the intermediates will only be in C++ PyTorch Tensors, all without the GIL. We have a small comment about it in our PyTorch book in the section on what improvements to expect from the PyTorch JIT and it seems rather relevant in practice.
I'll not advocate calling DigitalOcean on the phone, but tagging the CEO on the complaints on twitter might be more effective than tagging the community manager who professes he is not listened to.
Poll anything they might have a preference on (and not all the same) - movies, music, internet celebrities or what not - and build a system that asks someone's opinion on 5 things and then outputs suggestions.
I don't think you need anything fancy, but just counting "same entries" in the preference matrix would work to get basic collaborative filtering. Depending on the depth allowed, you can cast that more mathematically and maybe even discuss the idea of rank 1 factorizations (i.e. assign each person / thing a vector and attempt to approximate the matrix entries as scalar products) or optimization of that stuff.
As outlook, you have word2vec that works similar but has the added complication of the matrix being huge. Use that for a information retrieval or toy chat bot or whatever.
I did teach this to non-experts, but not high school kids. I imagine it would work similarly. Unfortunately, I don't know of a website that covers this for school kids.
If you know where to look (i.e. it's public but unannounced) you can see that nightly wheels have been built for the last few days. So I would expect that some time between now and the Developer Day in November we'll see ROCm appear on PyTorch's "get started" page.