Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?
I tried to present concepts in an as reasonably accurate mathematical way as possible, and in the end I cut through a lot of math in part to avoid the heavy notation which seems to be present in this book (and in part to make sure students could spend what they learnt in the industry). My actual classes had way more code than formulas.
If you want to write everything very accurately, things get messy, quickly. Finding a good notation for new concepts in math is very hard, something that gets sometimes done by bright minds only, even though afterwards everybody recognizes it was “clear” (think about Einstein notation, Feynman diagrams, etc., or even just matrix notation, which Gauss was unaware of). If you just take domain A and write in notations from domain B, it’s hard to get something useful (translating quantum mechanics to math with C* algebras and co. was a big endeavour, still an open research field to some extent).
So I’ll disagree with some of the comments below and claim that the effort of writing down this book was huge but probably scarcely useful. Who can read comfortably these equations probably won’t need them (if you know what an affine transformation is, you hardly need to see all its ijkl indices written down explicitly for a 4-dimensional tensor), and the others will just be scared off. There might be a middle ground where it helps some, but at least I haven’t encountered such people…
Apart from that, there is no “the code” equivalent. Mathematical notation is for stating mathematical facts or propositions. That’s different from the purpose of the code you would write to implement deep-learning algorithms.
Mathematical notation is really a shorthand for words, like you’d read text. The equals sign is literally short for equals. The added benefit, as others have pointed out, is that a good notation can sometimes be clearer than words because it makes certain conclusions almost obvious. You’ve done the hard part in finding a notation that captures exactly the idea to be demonstrated in its encoding, and the result is a very clean manipulation of your notation.
In hindsight, I think the issue was trying to map everything to programming is a bad idea and I was doing it because programming was the best tool in my tool chest. It was a real “when all you have is a hammer, everything looks like a nail” issue for me.
You might find it useful for your situation. The PDF is pay-what-you-want if you don't feel like paying for it.
I appreciate that some may find the book useful, but I personally don't agree with the presentation. There are too many conceptual errors in the book that you need to unlearn to make progress. For example, the book describes R^2 as a "pair" of real numbers. This is very much untrue and that kind of thinking will lead you even further astray.
I say this as someone with a math/cs degree and PhD having taught these topics to hundreds of students.
I naturally auto-corrected this "(the set of) pairs of real numbers". If that's the case, then I don't see how this differs from the actual definition. What is the conceptual error? Is it the missing 'set of'?
From page 15:
> The one piece of new notation is the exponent on R^2. This means "pairs" of real numbers.
Your interpretation of this quote is uncharitable at best. Using it to make a blanket assertion about the book is just silly, and quite out of the spirit of mathematics.
In particular, page 19 has an example of the kind of things that my book has that other books don't: a discussion of the soft skills of learning math and the cultural acclimation process:
> Though it sometimes makes me cringe to say it, give the author the benefit of the doubt. When things are ambiguous, pick the option that doesn’t break the math. In this respect, you have to act as both the tester, the compiler, and the bug fixer when you’re reading math. The best default assumption is that the author is far smarter than we are, and if you don’t understand something, it’s likely a user error and not a bug. In the occasional event that the author is wrong, it’s often a simple mistake or typo, to which an experienced reader would say, “The author obviously meant ‘foo’ because otherwise none of this makes sense,” and continue unscathed.
The course you suggested is the sort of "grab bag of topics" course, meant to cram the basics of every topic a CS major might want to know for doing the kind of CS theory research that MIT cares about. If you find math hard, I doubt that will make it much easier, but it could be good to do alongside a book like mine if you find my book too easy.
Arithmatic and writing proofs are very different skills. There is going to be a gap for everyone.
When all you have is a hammer and all that
And as such it is way too often abused. Because the (original, and the most useful) purpose of mathematical notation is to enable calculation, i.e., in a general sense, to make it possible to obtain results by manipulating symbols according certain rules.
This kind of book is more verbose than my liking both in terms of rigor and content. For example, they include Gronwall’s inequality as a lemma and prove it. The version that they use is a bit more general than the one I normally see, but Gronwall’s inequality is a very standard tool in analyzing ODEs and I have rigorous control theory books that state it without proof to avoid clutter (they do provide a reference to a proof). A lot of this verbosity comes about when your standard of proof is high and the assumptions you make are small.
I suppose the goal would be to understand deep learning so that we know enough of what's going on but not to get stuck in math concepts that we probably don't know and won't use.
I like this approach because it allows me to connect the math to the problem, whereas otherwise you wouldn't have.
In the book, you're slowly creating a DL framework, as the title says, from scratch. He also has all the code on GitHub: https://github.com/SethHWeidman/DLFS_code
I think if you are truly trying to understand deep learning, you will never get to avoid the math because that's really what it is at it's core, a couple of (non-linear) functions chained together (obvious gross oversimplification).
\left(\Psi_{L} \circ \mathcal{A}_{l_{L}, l_{L-1}}^{\theta, \sum_{k=1}^{L-1} l_{k}\left(l_{k-1}+1\right)} \circ\right. & \Psi_{L-1} \circ \mathcal{A}_{l_{L-1}, l_{L-2}}^{\theta, \sum_{k=1}^{L-2} l_{k}\left(l_{k-1}+1\right)} \circ \ldots \\
& \left.\ldots \circ \Psi_{2} \circ \mathcal{A}_{l_{2}, l_{1}}^{\theta, l_{1}\left(l_{0}+1\right)} \circ \Psi_{1} \circ \mathcal{A}_{l_{1}, l_{0}}^{\theta, 0}\right)
x_{\mathcal{L}(\Psi)+k-1} & =\mathfrak{M}_{a \mathbb{1}_{(0, L)}(\mathcal{L}(\Psi)+k-1)+\mathrm{id}_{\mathbb{R}} \mathbb{1}_{\{L\}}(\mathcal{L}(\Psi)+k-1), \mathbb{D}_{k}(\Phi)}\left(\mathcal{W}_{k, \Phi} x_{\mathcal{L}(\Psi)+k-2}+\mathcal{B}_{k, \Phi}\right)
In fact, this pdf is literally the resource I've been searching for as many others are far too ambiguous and handwavey focusing more on libraries and APIs than what's going on behind the scenes.
If only there were a similar one for microeconomics and macroeconomics, I'd have my curiosity satiated.
I recall giving Mankiw a try and wished I could just find a physics-style textbook as I found it way too wordy.
E.g., micro models are just constrained optimization based on the idea of representing preference relations over abstract sets with continuous functions. So obviously, the math is then very simple. This is considered a feature. You can also use more complex math, which helps with certain proofs (especially existence and representation).
You could grab some higher level math for econ textbooks, which typically include the models as examples, where you skip over the math.
For example, for micro, you can get the following: https://press.princeton.edu/books/hardcover/9780691118673/an... I think it treats the typical micro model (up to oligopoly models) via the first 50 or so pages while explaining set theory, lattices, monotone comparative statics with Tarski/Topkis etc.
I’m glad you’re enjoying the book. The approach is ideal for a very small subset of the ML population, no doubt that was their intention. I’m just weighing in that it’s entirely possible to cover this material with rigour yet much simpler notation. Even as someone who could parse this I’d go with other options.
Sure I might want some slight changes to the notation I found skimming through on my phone, but everything they define and the notation they choose feels pretty familiar and I understand why they did what they did.
Mirroring what someone else said, this is exactly the kind of intro I’ve been looking for for deep learning.
From what I’ve seen it’s a small percentage, and there’s no reason for most people to be put off by it.
Everyone come on in the water is fine.
I’m trying to think if it’s 0 percent outside of backprop…
Arguably high school math gets you quite a bit of understanding. After that in descending order I’d guess Linear Algebra, Statistics/Probability, Basic Calculus, Partial Derivatives…
In other words it’s not all or nothing. The easiest stuff gets you a lot of bang for your buck.
Screenshot the math, crop it down to the equation, paste into the chat window.
It can explain everything about it, what each symbol means, and how it applies to the subject.
It’s an amazing accelerator for learning math. There’s no more getting stuck.
I think it’s underrated because people hear “LLM’s aren’t good at math”. They are not good at certain kinds of problem solving (yet), but GPT4 is a fantastic conversational tutor.
What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4.
Also since it is fairly common for you are there any real world examples you can share?
Uhhh... it's like telling people to trust SO over reddit, especially a subreddit known to lie.
> What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4.
Both. Can we stop doing this? This is a fairly well established principle with tons of papers written about it, especially around math. Just search arxiv, there's a new one at least every week
There have not been tons of papers written about this.
You seem to be conflating papers about GPT4 as a solver with it as a math tutor. It’s a completely different problem space.
https://news.ycombinator.com/item?id=38845878
(older but similar) https://news.ycombinator.com/item?id=37904047
My main point consistently has been that GPT4 can be an invaluable resource specifically for learning math subjects.
I am not aware of any papers, studying people using it as a conversational tutor for learning math and having problems with hallucinations.
And?
> My main point consistently has been that GPT4 can be an invaluable resource specifically for learning math subjects.
This can also be true. I use it a lot. Don't confuse openly discussing limitations with calling it a pile of shit. No need to have only two extremes.
> I am not aware of any papers, studying people using it as a conversational tutor for learning math and having problems with hallucinations.
Very bad faith requirement. Unless you have good evidence that GPT hallucinates in many domains (as exemplified by said security report) and NOT math tutoring. If you have this really strong evidence that math tutoring is specifically unique then I suggest writing a paper. I'll help if you really can do it and be happy to give you first author and be proven wrong. But a much easier explanation is that math tutoring is not unique to GPT with regards of generating hallucinations. If you truly believe you do need a extremely specific example, you may need to pull the wool off your eyes. But I'm hoping you don't and are just arguing.
A lot of negativity comes from people who goofed around with 3.X for a while, came away unimpressed, muttered something under their breath about stochastic parrots or Markov chains that sounded profound (at least to them), and never bothered to look any further. 4 is different. 4 is starting to get a bit scary.
The real pedagogical value comes when you try to reconcile what it tells you about the equations with the equations themselves. Ask for clarification when something seems wrong, and there is an excellent chance it will catch its own mistakes.
And stop all this 3.5 vs 4 nonesense. We all know 4 is much better. But there's plenty of literature that shows its limits, especially around memorization. You also don't understand stochastic parrots, but in fairness, seems like most people don't. LLMs start from compression algorithms and they are that at their core. But this doesn't mean it is a copy machine despite the NYT article but it also doesn't mean it is a thinking machine like the baby AGI people. Truth is in between but we can't have a real conversation because hype primed us to just bundle people into two camps and make us all true believers. Just please stop gaslighting people when they say they have run into issues. The machine is sensitive to prompts, so that can be a key difference or sometimes they might just see mistakes you don't. It's not an oracle so don't treat it like one. And don't confuse this criticism as saying LLMs suck, because I use them almost every day and love them. I just don't get why we can't be realistic about their limits and can only believe they are a golden goose or pile of shit. It's, again, neither.
You have a parrot that can paint original pictures, compose original songs and essays, and translate math into both English and program code?
I would like to buy your parrot. I'll keep it in my Chinese room. There used to be a guy in there, but he ran away screaming something about a basilisk.
Kinda, kinda, yes, and yes.
I think there's far less originality than most people think. But it's not surprising when your job isn't leading you to look at thousands of pictures a day. I have yet to see a generative model that isn't pulling heavily towards the training data and you might be noticing the memorization rates are getting higher. But yes, a stochastic parrot doesn't mean memorization, it is about generalization and the stability around the p-norm ball around the training data.
Btw, what's wrong with a stochastic parrot? They are absolutely fucking useful. I use them every day. Hell, I even use things that are complete memorizations and all compression every day. What's with everyone equating powerful statistical systems with uselessness. Anyone saying that they aren't extremely useful is pulling wool over their eyes (but the same is true for anyone claiming baby AGI).
I'd also appreciate it if you discussed in good faith. The snarkiness is not appreciated.
https://i.imgur.com/JSWLFOi.png
I regularly get downvoted and criticized for suggesting this tool to other students, in defiance of what I can clearly see happening with my own eyes. I see a tool that, if developed further, will answer much deeper questions, including original ones, just as accurately and effectively. One that appears capable of taking humanity to the next level so fast it will make the monolith in 2001 look like an abacus by comparison.
Meanwhile, you tell me, "Don't suggest this to other students, it might hallucinate." Other people say, "Shut this down at once (or nerf it beyond any possibile utility), it might hurt somebody's feelings." Another contingent warns, "Shut this down at once, it might start a nuclear war." Still other people say, "Shut this down at once, it violates copyright law." The objections just get dumber from there, yet gain traction by the day.
There's never been a time when standing in the way of something like this was right. Why should I think it's time to do so now? (And yes, I acknowledge that you're not personally 'standing in the way', but it really bugs me when people who claim they aren't 'standing in the way' of the technology tell other people not to use it.)
I have yet to see a generative model that isn't pulling heavily towards the training data
When's the last time you saw a human mind that didn't work that way? (Or, for that matter, a parrot's mind.) The real truth behind the stochastic-parrot metaphor is that parrots, stochastic or otherwise, are nothing all that special, and neither are we. We're just better at using tools than the birds are, that's all.
Or at least we were up until now. But muh COPYRITE!!!11! ...
I think people in my camp (which often are confused with the Gary Marcus camp), aren't saying you shouldn't be blown away. Those people wouldn't say this
> And don't confuse this criticism as saying LLMs suck, because I use them almost every day and love them. I just don't get why we can't be realistic about their limits and can only believe they are a golden goose or pile of shit. It's, again, neither.
Fwiw, I give those people an ever harder time. They deny utility that is quite apparent. They also have these silly contrived doomer arguments that don't make any sense, as if one day AGI is just going to unexpectedly appear out of nowhere and, like you suggest, somehow jump the airgap and get control of the world's nuclear weapons without anyone noticing. What an insane hypothesis that doesn't have anything substantial evidence and is entirely based on "but what if!" It is conspiratorial and a distraction from the real harm these systems can do which is far more subtle and not really an existential crisis (at least arguably in the same way, but let's not get into that). Some of these people are shills and some are useful idiots/true believers. You're right to not pay attention to them.
I'll also mention that I too am blown away. But you can be blown away and still have criticism and be wary of a thing too. The answer is quite impressive, without a doubt. I mean we are literally putting lightning into rocks and making them capable of doing math and speaking human languages. If you're not blown away by any single one of those things then it is simply a lack of imagination.
> When's the last time you saw a human mind that didn't work that way?
Quite frequently. Same with even my cat, and she's dumb as shit. Probably ran into too many walls while chasing toys but I think that's just a feedback loop lol. She's dumb as shit but I'm also absolutely blown away by her brilliance. It may be hard to see that both those can be true, but that's the true state. But I disagree that there isn't anything special about stochastic parrots, any animal, or humans. They are all mind mindbogglingly impressive, just our brains are designed to normalize things to not be overburdened by the computational load (which itself is impressive!).
You are absolutely right though that there's a ton of exploitation that humans do (referring to exploration vs exploitation). I said memorization is incredibly useful. But creativity is far more subtle. I should put it this way, chimps (very impressive creatures), are far better at memorization than most humans. But they are nowhere near as creative. Certainly some creativity is leveraging prior works for inspiration. But a subtle aspect of this is that often when this form is considered brilliant it crosses domains, which is something no ML seems to even have the capacity to perform. This can be hard to know though because unless you have domain knowledge you may not have heard about how people like Einstein was called a mathematician and not a physicist or how Nash was said to "just used topology". This type of lore is important if we're going to discuss actual intelligence but not important for tools or our every day lives. The devil is in the details when we care about details.
It can be really hard to understand these distinctions. You have to look REALLY close at details. One thing I'll mention is that I know I have looked at the datasets we use in our group far more than anyone else that I know. This is unsurprisingly an uncommon thing because it is boring to look at the raw data and investigating things like LAION takes herculean efforts (something I haven't even approached). But your example is actually remarkably relevant to this topic. You couldn't have done anything better! Because most people rely on measurements of distance like cosine similarity or L2 to determine duplicates or near duplicates. But ask GPT this (you should get the right answer): "How does the curse of dimensionality relate to distance measurements in higher dimensions? Are there any problems this creates?" Or ask it another one, which even the fact blew me away the first time I heard it despite being absolutely obvious after I took just a moment to think about it: "If I have a n dimensional space, where n is very large, what is the expected angle between any two random tensors? What is it as n approaches infinity?" I'm positive it will again give you the correct answer.
But you also have to realize that this is frequently written about and without a doubt in the training data. You can absolutely overfit models and have them be incredibly useful. But the difference is that this won't be generalization and will be brittle. For a long time GPT was not able to correctly answer "Which weighs more, a pound of feathers or a kilogram of bricks" because it was too sensitive to the expected answer (it'll work now btw). It still has problems with a variation of the corn, goose, fox river crossing puzzle if you change it to allow all items in the boat at once (at least when I checked a month ago). But this is not the actions of sentient creatures. Ones that can think and comprehend. You're going to have to think really hard about how you think and especially how you think really hard to get a good understanding of this. But it comes down to the reason why someone can be absolutely brilliant while shockingly idiotic. This is not the quip from iRobot with the "can you?" about art and symphonies. There is something deeper and truth be told, many animals do things for no good reason (one that can't be clearly defined by our perceived loss functions, which may accurately be called emergent behaviors). Every mammal also is able to run complex simulations in their minds, at incredibly low computational costs. Even the small rat will twitch its legs while it sleeps or your dog may bark, being unable to distinguish reality from a dream, just as you do. That is truly a world model. Something we aren't remotely close to in AI, but that's okay. Why would it not be okay?
But in some way you are being gaslit, but not by what I intended to say (but maybe from how you read it. Which I apologize, I am trying to work on communicating better, but it is hard when we have a diverse global audience with many different base assumptions and knowing which type of imputation I need to direct my message at). There are plenty of people with highly invested interest to sell you these tools as far more than they are. I've written a few comments before that what's going on is as if we made a chocolate factory. One that sells the best god damn chocolate you're ever tasted. But then they started selling the chocolate as a cure for cancer. At that point, it doesn't matter how good the chocolate is, people will feel disenfranchised. Some people are responding by saying that the chocolate tastes like shit while others are saying it cured their cancer. But neither of these are true. It's damn good chocolate, but it isn't going to cure cancer. (ML certainly will be a very useful tool for tackling cancer. That was not the intent of this analogy) I just think there's this fear that people have that if something isn't a literal gift from god then it is a pile of shit, and I don't get it. Nothing we have fits that description but we have done and created so many incredible things as humans and made such leaps and bounds with these half baked incomplete things. There is nothing wrong with just okay chocolate, but the chocolate we have is without a doubt, better than just okay.
Does that make more sense?
Please be more original.