justinnk

This one [0] about the Higgs boson is hard to beat. Pages 25-32 are just full of around 5150 „authors“, pages 33-37 are their affilliations.

[0] https://arxiv.org/abs/1207.7214

Not sure whether Karpathy meant it like that or thought about this statement being interpreted this way, but I totally agree with you. Creating animated movies and special effects is placing a lot of 3d polygons by hand. Maybe not with JavaScript though. However, people also made things like

telnet towel.blinkenlights.nl

Now, were they „in their right mind“? I don‘t know. The more likely explanation is they found joy in it. Reminds me of what the Suno CEO Mikey Schulman said about making music: „[…] I think the majority of people don’t enjoy the majority of time they spend making music.“ (https://news.ycombinator.com/item?id=42688538)

I agree, this seems like an important discovery. And it hits a sweetspot of being effective and very simple to implement. Have a look at the pseudocode comparison at the bottom of the accompanying github page: https://explorative-modeling.github.io/

Also, a very well-structured and easy to follow blog post by the author. I very much enjoyed reading it!

I once heard a lecture about „learning how to learn“. There are already a lot of well-researched things one can exploit about the way we process information. What you describe is known as retrieval cue. Our minds seem to be associative^1. It also seems a lot of the learning/processing happens after the actual learning process, which is why resting is important. Spaced repitition is a further method well-proven in practice.

^1 In the early days of AI, the concept of Hebbian learning was popular („neurons that fire together, wire together“). Its implementation is even simpler than gradient descent used today, but never caught on https://en.wikipedia.org/wiki/Hebbian_theory

Exactly. If someone works on bioengineering viruses that could start a global pandemic, they have to ensure a highly secure working environment. Nothing must ever escape the lab unintentionally. It’s basically common sense. Similar standards should be held when doing such experiments with computer programs that are capable of causing global damage. It must physically be impossible to send anything to the internet.

Interesting analogy. I believe regarding addictiveness they may be compared.

> a shift of skills away from things that mattered more in the past toward other things that are not measured/perceived by the older generation.

Do you have any ideas what these things might be? As someone in his twenties, I’m sometimes saddened by observing that some of the skills I acquired over a long time (e.g., writing, coding) may become obsolete or won’t be respected anymore just now that I‘m finally getting good at them.

Totally agree that it requires struggle and I did not say you should use it to just get the solution. What I think one can do is use it more like a personalized textbook which you can ask any question. It can also provide you with problems just at the right level for you and judge the solution. Now, of course for many students it is tempting to just get the solution if provided with the means but they can be taught to use an LLM in a didactically useful way.

I see where you are coming from.

However, in the role of personal teachers they may allow especially our young generations to reach a deeper understanding of maths (and also other topics) much quicker than before. If everyone can have a personal explanation machine to very efficiently satisfy their thirst for knowledge this may well lead to more good mathematicians.

Of course this heavily depends on whether we can get LLMs‘ outputs to be accurate enough.

They already had a basic form of this for a while [1]

> arXiv requires that users be endorsed before submitting their first paper to arXiv or a new category.

[1] https://info.arxiv.org/help/endorsement.html

I think it really depends on the how. Engaging with it in a socratic debate-style argument [1] if no fellow human is available might very much support your thought process. On the other hand, just obtaining the solution to one‘s homework/problem/task/… won‘t be very beneficial for one’s development. The latter is sadly much more convenient and probably accounts for most of the usage. I remember a saying about the mind being a muscle: in order to keep it in good shape, you have to use it actively.

[1] https://en.wikipedia.org/wiki/Socratic_method

Thank you for the good laugh! This whole thread is peak satire. Although, be careful. It reminds me of the foreword to a shortstory someone shared on HN recently: „[…] Read it and laugh, because it is very funny, and at the moment it is satire. If you’re still around forty years from now, do the existing societal equivalent of reading it again, and you may find yourself laughing out of the other side of your mouth (remember mouths?). It will probably be much too conservative.“ — https://www.baen.com/Chapters/9781618249203/9781618249203___...

I believe Enzyme comes close to what you describe. It works on the LLVM IR level.

https://enzyme.mit.edu

What you suggest seems plausible, but there is a very good counter example. Overleaf is also managing well by relying on the open-source LaTEX. What drives people to subscribe is not the typesetting itself, but the ecosystem around it (collaborative editing, version management, easy sharing, etc.). You can make money with those and still have the rendering free/open-source. I believe a similar thing is/will be true for Typst as well.

Reminds me a bit of Isaac Asimov‘s novel „I, Robot“ where they rely on positronic brains to do things. In the story, mathematics seems to have caught up and developed a framework to analyse the behavior of an AI system. I wonder if something similar will happen if CS becomes an empirical science, i.e., will we try to infer laws from empirical AI behavior measurements so that we can reason about it more effectively? This would then turn CS into Physics somewhat, but based on an artificial system. Very strange times.

> these AI systems will be flying our airplanes, running our power grids, and possibly even governing entire countries.

I guess we should figure out how to include the three laws of robotics in connectionist models asap…

I can second this, even availability of the code is still a problem. However, I would not say CS results are rarely reproducible, at least from the few experineces I had so far, but I heard of problematic cases from others. I guess it also differs between fields.

I want to note there is hope. Contrary to what the root comment says, some publishers try to endorse reproducible results. See for example the ACM reproducibility initiative [1]. I have participated in this before and believe it is a really good initiative. Reproducing results can be very labor intensive though, loading a review system already struggling under massive floods of papers. And it is also not perfect, most of the time it is only ensured that the author-supplied code produces the presented results, but I still think more such initiatives are healthy. When you really want to ensure the rigor of a presented method, you have to replicate it, i.e., using a different programming language or so, which is really its own research endeavor. And there is also a place to publish such results in CS already [2]! (although I haven‘t tried this one). I imagine this may be especially interesting for PhD students just starting out in a new field, as it gives them the opportunity to learn while satisfying the expectation of producing papers.

[1] https://www.acm.org/publications/policies/artifact-review-an... [2] https://rescience.github.io

This is very interesting! I think an exciting direction would be to arrive at minimal circuits that are to some extent comprehensible by humans. Now, this might not be possible for every system, but certainly the rules of Conway‘s GoL can be expressed in less than 350 logic gates per cell?

This also reminds me of using Hopfield networks to store images. Seems like Hopfield networks are a special case of this where the activation function of each cell is a simple sum, but I’m not sure. Another difference is that Hopfield networks are fully connected, so the neighborhood is the entire world, i.e., they are local in time but not local in space. Maybe someone can clarify this further?

You can actually use this to import pdfs generated with Matplotlib as vector graphics into Impress presentations. This allows you to change, e.g., the color of lines or the legend (or any other part of the plot) right within Impress to better fit your presentation. I found this extremely useful in the past. In Powerpoint, I could not even import an svg, let alone a pdf (although maybe the newest version supports this?). The only downside is that currently you have to first import the pdf into Draw and then copy the shapes/curves over to Impress. I hope they will add direct import into Impress in the future.

There is also an open-source/free version of this [1], which I use regularly. You can install it, e.g., in Fedora, with the ‚diffpdf’ package. It is no longer maintained but works very well, has a nice GUI with a side-by-side view, drag&drop support, and both text and visual modes.

[1] https://www.qtrac.eu/diffpdf-foss.html

(I am one of the authors) Generally speaking, the latter. The purpose of DiscoGrad is just to deliver useful gradients. These provide information about the local behavior of the cost function around the currently evaluated point to an optimizer of your choice, e.g., gradient descent. Interestingly, the smoothing and noise can sometimes prevent getting stuck in undesired (shallow) local minima when using gradient descent.

Thanks! You may find DeepProbLog by Manhaeve et al. interesting, which brings together logic programming, probabilistic programming and gradient descent/neural networks. Also, more generally, I believe in the field of program synthesis there is some research on deriving programs with gradient descent. However, as also pointed out in the comment below, gradient descent may not always be the best approach to such problems (e.g., https://arxiv.org/abs/1608.04428).

You are right in that the use-cases are very similar to regular autodiff, with the added benefit that the returned gradient also accounts for the effects of taking alternative branches.

Just to clarify: we do a kind of source-to-source transformation by transparently injecting some API-calls in the right places (e.g., before branching-statements) before compilation. However, the compiled program then returns the program output alongside the gradient.

For the continuous parts, the AD library that comes with DiscoGrad uses operator overloading.

(I am one of the authors) Thanks for your question. Yes, similar to what you describe but not quite. The prime use case is to apply DiscoGrad together with a gradient descent optimizer to optimization problems. For a C++ program to be regarded as such, you have to define what the "inputs" are and the program has to return some numerical value (loss) that is to be maximized/minimized. The tool then delivers a "direction" (smoothed gradient), which gradient descent can use to adjust the inputs toward a local optimum.

So if you can express your test cases in a numerical way and make the placeholders for the "magic numbers" visible to the tool by regarding them as "inputs" (which should generally be possible), this may be a possible use-case. Hope this clarifies it.