A First Course in Differential Equations for Scientists and Engineers | Hacker News Reader

A First Course in Differential Equations for Scientists and Engineers

people.uncw.edu

514 pointslainon65 comments

While the linked course seems a lot more thorough, I took the Udacity "Differential Equations in Action" [1] course, which I found very well done. For the homework you write Python programs to compute things like gravitational slingshots, modeling epidemics, wildfires, and the n-body problem.

[1] https://www.udacity.com/course/differential-equations-in-act...

While the linked course notes (I looked at the pdf, definitely good textbook quality and not just course notes, really) are absolutely more thorough in the classical undergrad diff.eq. topics, the Udacity course touches some further topics (stiffness, implicit methods, control theory) that the course notes don't.

Can we get to the text and videos without all the school-ish crud? (starting a course, timelines etc.)

Yes [1]. I haven't taken a Udacity course in quite a while, but at the time all of their videos were hosted on Youtube, so you should be able to find a lot of their courses just on Youtube. For the homework, I remember there was an include file that they didn't provide the source code for, but a Google search turned up a version someone wrote for using it locally. Usually it's pretty obvious when you've solved the problem and have the correct code, but to make absolutely sure, you have to upload your code to their server for verification. But you can take the whole class without ever visiting Udacity.

[1] https://www.youtube.com/watch?v=8cLXVG2Q6D4&list=PLAwxTw4SYa...

thanks!

I took the first part of that course, enough to write an n-body game:

https://github.com/madengr/nbody

While it was a good course, I wish it covered the analytic portion, rather than dive straight into numerical solutions.

That is pretty neat. Improvements on the analytic side (if you are interested in the opinion of a random person on the internet) would be to scale your coordinate system (e.g. scale the distance and time to make average velocity and radius equal 1) and look into using a symplectic integrator. Better for these types of systems.

Thanks. I remember in the course, keeping the energy constant is the big issue. They used the symplectic integrator to improve it. In the game, if you let it run long enough, the error visibly shows up.

Speaking of differential equations, an oldie (1997) but goodie, "Ten Lessons I Wish I Had Learned Before I Started Teaching Differential Equations" by Gian-Carlo Rota: https://web.williams.edu/Mathematics/lg5/Rota.pdf

> 4. Teach Changes of Variables

As someone who has basically no formal training in mathematics outside what's required for undergraduate computer science this is a big one. The first time I saw it I was blown away. Everyone who knows of it seems to treat it as natural as breathing, and not worth the exposition

I took DiffEq as an EE undergrad and, to my surprise, aced the course. For some reason, the concepts just resonated with me unlike other math courses where I had to work to gain an understanding of the material. A few years later, after graduating, I was taking courses leading to an MSEE and Partial DiffEq was offered as an option that I took. Thinking it was going to be similar to regular DE, I was soon in way over my head. Totally different material and concepts. I did end up passing the course but I can honestly state that I don't think I deserved it.

We used the Boyce and DiPrima book back when I was an electrical engineering student (Elementary Differential Equations and Boundary Value Problems https://www.amazon.com/Elementary-Differential-Equations-Bou...).

I remember it was an excellent book with many great examples and correlation with physics topics like mechanics, waves etc. Too bad our other mathematics books weren't at this high standard :/

Also, I really don't think that such topics can be self-studied. It seems really difficult to me to understand such topics without a teacher!

> I really don't think that such topics can be self-studied

Can you explain this? What makes something extremely difficult to be studied without a teacher? (I took calc in high school and never took diff eq, so my knowledge in this specific domain is basically zero)

If you have some mathematical maturity (you can read notation, work to understand each equation, have the patience to pretty much understand each equation and each page, because later ideas are built on understanding the previous ideas first), you can self-study a lot of mathematics. But almost everyone (mathematicians, physicists, engineers) who has that maturity, has probably taken a first course in differential equations during their undergrad studies.

So the number of people who have developed some mathematical maturity without ever taking a diff.eq. course is probably small. So we don't know much about these people and how the topic of differential equations appears to them.

But yes, it would be interesting to hear about experiences from someone with e.g. a strong background in theoretical computer science, probably including calculus but not including diff.eqs., who has self-studied a book on differential equations.

Just chiming in to say we do indeed exist and your example is pretty spot on, but I unfortunately have not self-studied diff.eqs. if I eventually do, I'll make sure to write something up about it.

I'd like to think I have developed a hint of mathematical maturity (CS + late declared Math major), but I went more the algebra route and ended up never taking a diff.eq. course. The closest I came was in a complex analysis course with some motivating examples that assumed we'd have picked up some tricks in a diff.eq. course.

I had self-studied this and then later studied under a teacher as well.

I think the thing that I found hardest in my self-study was (and unfortunately this is about 25 years ago so my recollection might be a bit off) that it seemed like there was a lot written on just two equations (the heat equation and the wave equation). I didn't get why is 50 pages dedicated to one equation. Up until that point it felt like Calculus was about techniques to solve equations, and then it suddenly became mostly about how to solve these two equations (there was a third, but I can't recall what it was now), which never really resonated with me.

Heat equation / diffusion equation (same thing): parabolic.

Wave equation: hyperbolic.

Laplace's equation: elliptic. Laplace's equation is the steady state (equilibrium) solution of heat equation when the boundary conditions (or anything) don't change in time.

Thanks. And yes, Laplace was the third one. I think even your simple categorization would have made things much simpler for me. Or maybe I was just an idiot as an undergrad. :-)

With calculus, you are already well into ordinary differential equations -- partial differential equations are different, but with calculus you have a good start on the basics of those, too.

A simple ordinary differential equation is below where of course just from calculus

y'(t) = d/dt y(t)

and the equation is

y'(t) = k y(t) ( b - y(t))

So, for the context: t is time, say, in seconds. y is some real valued function of t, that is, y(t), b and k are constants. We are given the value of y at 0, that is y(t) at t = 0, that is, y(0). We want the value of y(t) for t > 0.

Okay, with that, just need to use freshman calculus, for positive time s, integrate y'(t) from 0 to s. This is a simple exercise, without looking up the last dozen times I did that, maybe use integration by parts or some such. End up with a quotient with some exponentials.

Differential equations pop up in motion, e.g., from Newton's second law, AC circuit theory, and some other areas of science and engineering.

Boundary value problems, e.g., vibrating stings, parts of deterministic optimal control, are closely related but, still, significantly different.

Long some of the pure mathematicians went, in a word, "nuts" studying differential equations. The best of the results are good, and some of those are nicely useful. In places the work has nice contact with linear algebra, matrix theory, and vector spaces of functions, functional analysis, e.g., Hilbert and Banach spaces. But hanging over the whole subject is a suspicion that, really, as nice as the general theories are, mostly the applications are just a few, standard differential equations. It's a little like learning everything about civil engineering when really are only going to do framing carpentry, hang drywall, and apply roof shingles.

Once I bought

Garrett Birkhoff and Gian-Carlo Rota, {\it Ordinary Differential Equations,\/} Ginn and Company, Boston, 1962.\ \

I looked through it, saw lots of intricate stuff, but wondered just why I should dig into that. Since then I read a story about Rota about how, apparently, he felt much the same about the material, got stuck teaching the differential equations course because he wrote that book, and wanted, essentially, to f'get about that book and its material!

I had a full college course in ordinary differential equations. Okay: It left me wildly over educated for the differential equations in AC circuit theory. Otherwise I didn't much like the book, the teacher, or the course.

On the advanced stuff, here is some more

Earl A.\ Coddington and Norman Levinson, {\it Theory of Ordinary Differential Equations,\/} McGraw-Hill, New York, 1955.\ \

It has a nice result of Caratheodory, but in general could lose a lot of sleep working through that!

I had a course from a Ph.D. from MIT from the book, apparently long a standard at MIT,

Francis B.\ Hildebrand, {\it Advanced Calculus for Applications,\/} Prentice-Hall, Englewood Cliffs, NJ, 1962.\ \

So, yes, can find out about solutions via infinite series and boundary value problems. The book was very short on proofs, and to take such material seriously I wanted to see the proofs. Now that I know a lot more math, no doubt some of it originally motivated by material in that book, maybe I could fill in the proofs.

When I was at FedEx, I wondered about the cheapest way to climb, cruise, and descend the airplanes, had heard about

Michael Athans and Peter L.\ Falb, {\it Optimal Control:\ \ An Introduction to the Theory and Its Applications,\/} McGraw-Hill Book Company, New York, 1966.\ \

and flew up to MIT and met with Athans, got his course notes, etc. He explained that an application would be a "two point boundary value problem with mixed end conditions" -- okay, I'd had a course on numerical methods for that. But, in the early parts of the book will see something interesting -- fast, and well written coverage of the differential equations material needed for the book. This is an example of a general situation: Sometimes the best place to learn something is in an introduction or appendix written by a real expert who is also a good writer, intended as background for the rest of the book. So, such a source cuts out the tangential, maybe curious cruft can't much hope to use.

At one point after college on my own I carefully read, not nearly new at the time (TeX markup):

Earl A.\ Coddington, {\it An Introduction to Ordinary Differential Equations,\/} Prentice-Hall, Englewood Cliffs, NJ, 1961.

Coddington was not just a grand expert in the field but also a good writer. I really liked his stuff on variation of parameters -- a bit amazing. Note: Can find mention of that in the famous movie The Day the Earth Stood Still -- apparently that math was hot stuff in applied math about when the movie was made.

Can say some quite similar things about partial differential equations -- e.g., there are deep books, some connections with functional analysis (and even the theory of distributions) but the main interests are the partial differential equations of mathematical physics, especially, Maxwell's equations, the heat equation, the wave equation, Schrödinger equation, a wave equation, and the notorious Navier-Stokes equations -- which likely should attack only for limited goals and in somewhat special cases.

Net, unless you have some significant reason for more, I suggest you learn what you need to know, just in time, when and if you need it. But, in that case, as elsewhere, a good pure math background in calculus, and advanced calculus with the proofs, etc. will be good to have.

> I looked through it, saw lots of intricate stuff, but wondered just why I should dig into that. Since then I read a story about Rota about how, apparently, he felt much the same about the material

Ah yes, the classic rant by Rota. Entertaining read, and good perspective (10 pages). Apparently written in 1997.

https://web.williams.edu/Mathematics/lg5/Rota.pdf

Btw, the third formula on p. 9 is slightly wrong. Don't anyone lose your sleep over it.

> hanging over the whole subject is a suspicion that, really, as nice as the general theories are, mostly the applications are just a few, standard differential equations

I see something similar when it comes to machine learning. If you start to really dig into the underpinnings of the topic, you find fairly complex things like partial derivatives (for gradient descent optimization, for example), but you don't really have to understand it much to take it, apply it, and verify that the results make sense. On the other hand, I've been around long enough to have learned the hard way that applying something you don't really, fully, from-the-ground-up comprehend can bite you in surprising ways.

I'd suggest learning about partial derivatives and in particular, the gradient. With an appropriate book, one evening should be enough. Intuitively the negative of the gradient is the direction to ski fastest downhill, and that's why it is heavily involved in optimization problems. Then there are close connections with convexity -- intuitively the inside of most kitchen cereal bowls is convex. The gradient is crucial in uses of Lagrange multipliers in the non-linear cases; in non-linear optimization with constraints, the gradient is central to the Kuhh-Tucker-Karush necessary conditions for optimality. If are in a deep, long, narrow river valley, want to get to the bottom downstream of the river, and ski downhill using the gradient, then will keep crossing the river over and over, traveling many feet across the river for each foot going downstream. So can approximate the valley with an ellipse and ski in much better direction along the long axis of the ellipse. People figured this out long ago -- it's called conjugate gradients. If the river wanders, then that's still more difficult. In the best fitting in ML, may be in such river valleys, and some notes on ML recognize this and warn about using just the gradient. A lot more with gradient is known and at times useful -- Newton iteration, quasi-Newton, etc.

Vigorously seconded. There is a lot of applications of convex analysis, convex duality, KKT conditions and game theory in ML if one looks at it right. In fact the thats at the very foundation of techniques such as support vector machines (large margin separators in a Hilbert space), regret minimization algorithms, etc etc.

Of course one can choose to ignore all that and only focus on stochastic gradient descent. That will carry one for some non-trivial distance.

> If you start to really dig into the underpinnings of the topic,

Let's cover the most important part of partial derivatives, the geometric intuition.

Imagine the Smoky Mountains of east Tennessee, that is, smooth, rolling hills. Now to represent this landscape in math, let R be the set of real numbers, R^2 pairs of real numbers, that is, the coordinates of points in the plane with orthogonal axes X and Y, and let f: R^2 --> R, that is, f is a function of two variables, say, x and y, that is, the pair (x,y) in R^2, and the value f(x,y) is the height of the mountains above point (x,y), that is, a plane under the mountains.

Then at a point (x, y) the partial derivative of f(x,y) with respect to x is just the slope as in ordinary derivative of the mountain at point (x,y) in the direction of changing x. So, if the X axis runs east and west, the partial derivative of f(x,y) with respect to x is the slope of the mountain at (x,y) in the east-west direction. So the partial derivative is just like the derivative of a function of one variable, that is, a slope, except is for just one variable, say, x, with the other variable(s) y held constant.

So, if

f(x,y) = 3xy + 2x - y

then the partial derivative of f(x,y) with respect to x is just

D_x f(x,y) = 3y + 2

and the partial derivative of f(x,y) with respect to y is

D_y f(x,y) = 3x - 1

These partial derivatives are important in vector analysis and, thus, Maxwell's equations, electro-magnetism, fluid flow, optimization, etc.

Diff. eq. is a difficult topic. Not only you need to have a good understanding of a lot of other mathematic topics (derivatives, integrals etc) to properly understand it since it builds upon those but you also need somebody to guide you on how to think when solving such problems (i.e you'll need to solve diff. eq. together with your teacher).

Yeah - I never studied differential equations in college; I sort of know what they are, and that there's a famous problem that involves a snow-plow plowing a field while it's still snowing, but that's about it. I've always been curious, though, so I clicked through the link. Per the introduction, I jumped ahead to the "what do you need to know from calculus" appendix just to see how much I did remember. Derivatives, integrals, ok, sure I remember that. Trigonometric functions? Um, yeah, I think I remember that - adjacent over hypotenuse, tangent, theta, sounds familiar. Half-angle, double-angle... getting rustier, but I think I remember that stuff. Hyperbolic functions? Oh, boy... yeah, something to do with the way power lines hang. I remember remembering that once. Integration by parts? Oh, man, I don't know if I ever learned that. Geometric series? Power series? Binomial expansion? I read about those in TAOCP, but if I ever studied those in class, it went in one ear and out the other. And that's what I need to know to start reading... maybe I do need a teacher.

Ugh, I hated that book. It never went into any detail in its “solutions” and the whole thing seemed like a bag of tricks for solving whatever example they were currently showing. Plus I had to deal with the monstrosity that is WileyPLUS.

ODEs as taught in a first-course undergraduate course is precisely a bag of tricks.

That you will never use in real life.

The chapters are first, second, and higher order were all a bunch of useless tricks. Then they introduce you to series solutions, where it starts to feel useful, and then they hit you on the side of the head with useless transforms.

ODEs are taught the way they are taught out of tradition, a bad tradition.

Those useless transforms (are you referring to Laplace and friends?) are pretty useful in EE.

Yep, the transforms and series solutions are actually the most important part of ODEs, not just in EE but also in PDEs.

All fair criticisms, but I was recollecting on reading Boyce & DiPrima where the Laplace was relegated to IVP.

I was left with feeling that I had only learned yet another trick until I read Churchill & Brown's Complex Variables and Applications.

Indeed, and lame ones at that. I've found the qualitative theory of ODEs infinitely more interesting and useful for working with real problems.

I couldn't disagree more about needing a teacher. While I did take DiffEQ in college with a "teacher", his handwriting was too atrocious and his polish accent was too thick that I ended up learning the entire course myself.

To be fair though, as a physics major we were already dabbling with some of the stuff in other classes.

Look at Arnold's book on Diff, it is harder then others, but has absolutely different outlook.

That book is totally mindblowing, and obliterates artificial boundaries between physics and mathematics. It also (in the older Dover editions) had a cover where the phase portrait on the front looked like two angry eyes glaring at you that you hadn't learned enough math yet.

Well, modern theoretical physics seems to be nothing but mathematics (mostly advanced differential geometry and group theory). And this is a good thing, as it is the sign of how far along the subject has gone in its evolution. The theory of differential equations also reduces to differential geometry, which is what Arnold’s book seems to be striving to show, as does his other excellent text on mathematical principles of classical mechanics.

I can only find the first edition on Amazon, but there's apparently a third edition? http://www.matem.unam.mx/ernesto/EDO/Ar-EDO.pdf

https://www.amazon.com/Ordinary-Differential-Equations-MIT-P...

I think this is the third edition: https://www.amazon.com/x/dp/3540345639

On a related note, I can recommend Stanley J. Farlow's "Partial Differential Equations for Scientists and Engineers", which I bought around 2004 to better understand a fluid simulation paper I was trying to implement. (My Maths course ended with linear algebra and ODEs). Very good read with nice physics examples.

IMO, Paul's notes http://tutorial.math.lamar.edu/Classes/DE/DE.aspx are a lot easier to use.

The usage of terms 'initial value' and 'boundary value' is a massive failure of mathematics education.

'Initial' refers to time-like variables, and 'boundary' refers to space-like variables.

There is no notion of time in mathematics. There is only a notion of space, due to geometry. I had the longest time coming to grips with the question, "mathematically what is the difference between initial value and boundary value?" only to realize the distinction is meaningless in mathematics. It's a relic of the past when differential equations were studied under physics, where time and space are a huge part of the conceptual foundation.

Sometimes I wonder how much progress we would make in education if we didn't confuse the heck of our students in the name of convention and historical baggage.

Since time is 1D while space may have more dimensions, initial values are connected to ODEs while boundary ones with PDEs. Their behavior is quite different.

Huh... didn't expect to find a UNC-Wilmington link on HN today. I vaguely remember Dr. Herman. I think we used one of his other books in a class, or maybe he taught one of my classes. This brings back memories.

This was my favorite course in college. Super beneficial materials.

On a side note, one of the reasons I loved this course was because it was online and only had 2 tests with no other assignments. The professor allowed you to schedule office hours any time you needed, but the course setup was sweet for self-studiers like me. Here's the book, Chapters 1-10 are on the midterm, Chapters 10-20 are on the final. No homework busy work, no other tests. Just 2 exams. Go.

When I was in college I supplemented my learning with Notes on Diffyqs, which I thought explained things much clearer and concisely than my assigned textbook. It is available for free and has tools for modeling systems in Octave/Matlab.

https://www.jirka.org/diffyqs/

On a related note, how do you take notes for math subjects with their multiline integrals and sumation and subscripts, division etc?

I'm very much attached to recording every thing in simple text editors. Is there a "Notepad" or "TextEdit" for mathematical notation?

you can use LaTeX notation and then render it with something.

For example MathJax, or emacs's org-preview-latex-toggle to show the rendered equations in the same buffer you write in.

Don’t. Pay attention, write down the things that aren’t in the text book. And then learn to read the text book. It’s a hugely valuable skill.

I'm easily distracted, particularly from internal digressions. Taking notes helps me hold on to the thread.

I could not imagine trying to take notes with a computer in a math heavy subject. I had to use paper for everything during my undergrad in Civil Engineering. However, if you already have a computer with you, record the lecture and review it later.

Just as a partial counterpoint to this and overall agreement with everyone else who responded, I've had success taking realtime notes on my laptop in a math class. It took a bit of practice and I'm sure I can't do it anymore, but it's not insurmountable by any means.

I used latex and made liberal use of keyboard and software macros to do it, and one of the tricks was to realize that if I needed a quick-to-type way to typeset new thing X, I should just pretend I had such an implementation and make up its command on the spot. At my leisure, I could write up a conforming latex command that worked with all the notes I'd taken in realtime.

That said, I've since come to realize that math notes don't help me as much as they seem to help others. I have greater success primarily listening during class and leaning on the textbook as well as online resources outside of class. I do second the use of emacs to handle the latex, but I don't think that realtime rendering is particularly important in a notes setting.

Excellent counterpoint. Keep in mind that I was a Civil Engineering student, not computer science. My abilities to use Latex were little-to-none at the time. I didn't get any exposure to it until I was in grad school and we used it to format journal article submissions.

I use mathstackexchange for that purpose and do all my work in the little window where they want you to type up your question. I don't ask/answer questions on there. Mathjax will render all your stuff beautifully. If I want to save my work, I just take a screenshot.

This link below is a great reference that's worth keeping around:

https://math.meta.stackexchange.com/questions/5020/mathjax-b...

Not exactly what you're looking for but a near fit: http://www.wolframalpha.com/input/?i=derivative+of+x%5E4+sin...

Do not deprive yourself of the delight of browsing through your handwritten notes some years in the future!

I've been playing around with: http://asciimath.org/

There's TeXmacs, which lets you enter mathematics either via point-and-click palettes or using TeX notation. I like it a lot. Downsides: 1. Development seems more or less dead. 2. It's not always been perfectly stable. 3. It's kinda sluggish, especially on older slower machines.

Lately I have been looking for a practical (not too much theory) book on how to model problems with differential equations, with lots of examples. I imagine I'd plug them into Wolfram Alpha to get a solution.

Does anyone have recommendations?

Almost all textbooks named "Mathematical Modeling" should have chapters on modelling continuous change with differential equations, hopefully also a chapter on how to model systems of interacting variables with a system of differential equations.

Then also textbooks with names like "Mathematical Models in Biology" or "Mathematical Biology" should have chapters on population growth, a two-species predator-prey model, diseases and epidemiology, and sometimes also a chapter on chemical kinetics (these are all modelled with differential equations).

Thank you. Knowing the right search terms will help a lot.

You can also look into "system dynamics" software, which allows for the modeling of complex systems with very little user-facing mathematics. Dana Meadows wrote a book called "Thinking in Systems," which is a great intro to this subject.¹

I personally have used Stella by isee systems, which will output an equation from your model if you are so inclined, but I think there is cheaper/free software out there that will do similar types of modeling.

1. https://wtf.tw/ref/meadows.pdf

Thanks, that looks very interesting!

Collection of PDFs, more/less like a textbook. Looks worthwhile.

Reply on news.ycombinator.com