Python Programming and Numerical Methods: A Guide for Engineers and Scientists
pythonnumericalmethods.berkeley.edu
pythonnumericalmethods.berkeley.edu
The Python community has done an amazing job making a suite of scientific software. Numerical libraries and plotting are excellent in Python. Sympy and Sage are both very ambitious, and in my opinion, quite successful projects. So what’s the problem?
Longevity and stability. Make fun of it all you want, but old crusty FORTRAN code, and in the past 20 years C and C++, are the bedrock of serious scientific applications. Python usually isn’t. I’m not confident enough to say why with certainty, but maybe it has to do with the fact that Python APIs and type signatures (whether written out or not) are too handwavy and easy to change without detection.
In addition to this, I rarely find that people, scientists especially, are writing applications in Python. Python is too hard to deploy as an application sane people can use, is a mess to configure, and difficult to make efficient if your slow parts aren’t rewritten. Most scientists slop code around in Jupyter notebooks which bitrot and are forgotten about in 6 months.
The world is clearly in need of good numerical code and environments, even with Python’s amazing ecosystem. Julia has been popular among my savvier programming colleagues and it has attracted very high quality contributions, rivaling and exceeding Python in some cases.
Back to me: what do I do? I write my numerical applications in Common Lisp. It’s probably as unpopular of a choice as it gets in the scientific world. I use Lisp because:
- The language is standardized and my code won’t break when the compiler updates
- I can write very high performance floating point code, using SIMD/AVX, and readily inspect the assembly code at the function level
- I can easily bind to C and Fortran libraries
- I can deploy applications as static executables
- I can avoid garbage collection and heap allocations with careful programming
- I can write generic, extensible interfaces
- I can debug algorithms live—in the depths of stack traces—without the compile-printf-edit cycle, which is insanely useful when encountering tricky floating point accuracy bugs.
Clearly I’m a “programmer’s programmer” though, which may be a hint as to why Lisp remains unsuitable for scientist use. Most scientists don’t know or don’t care what a stack allocation is, and that’s fine, because the tools have evolved around them in such a way they can be hugely successful without that knowledge. It’s also remiss to not mention that scientists typically have 8+ years of training in their (non-software) discipline, and “ain’t nobody got time for [programming],” despite its growing importance and necessity.
The biggest negative of using Lisp is that a lot of work you’d take for granted in Python or Julia is not done. It took me several hours to get a certain complex 2^n dimensional matrix factorization routine working from LAPACK in Lisp, whereas in Python it would take me 30 seconds flat to look it up, read the documentation, import it, and call it. So it’s definitely the case that Lisp’s “startup overhead” is high. But at the end of the day, if I’m building an electromagnetic simulator, I feel some pre-baked math routines are the least of my concerns, and the overall code structure and quality is of the highest concern. Python is suboptimal in that category, in my opinion.
One good thing about college courses like these is that they (slowly) displace numerical programming courses that are MATLAB tutorials in disguise.
The reasons are simple:
1. Lots more help and examples available online for python compared to FORTRAN.
2. Python is performant enough, 99% of the time if you make even a vague attempt to appropriately vectorize your code and/or use the right libraries and techniques.
3. Solving a well defined problem in any language is easy enough. What's hard is getting to that well defined problem, and that often involves playing and tweaking with things until you wrap your head around the problem space. Python saves me a lot of time so I can iterate faster by avoiding stupid errors and having much lower boilerplate overhead where stupid errors can really propagate into hard to follow error messages down the line.
Python just is a lot more intuitive. I don't have to waste nearly as much time on off-by-one and other stupid errors because my indexing was fucked up. So I can spend most my time on thinking about the stuff that really matters rather than implementation details.
That said, I can write some lean and mean FORTRAN if I really need to, and I'm ok with c++ when I need to be too. In reality though, most of my workload isn't that computationally intensive, and even when it is, most of the hard parts have been outsourced to c++ behind the scenes anyway. I can't even remember the last time when my computational need was bad enough that I considered writing my own fortran.
Microcontrollers are a different story. Trying to use python for that seems like a dumb idea, but I know MicroPython is a thing, though I'm skeptical of the whole concept to be honest. You're so close to the metal at that point that why would you want to abstract it away?
Looking back over the span of 4 decades, one big change is that there is just more stuff, and the specific programming language is just a small part of it. An advantage of learning something like Python is that you can do more with it, quickly, thanks to the libraries.
Julia is great in this regard, but it seems that over time, any language with a centralized library repository will become a graveyard for unmaintained legacy code, and this is bad for long term scientific use.
The downsides of rolling your own mathematical functions is that it's very time consuming, and there's always a chance you'll get it mostly right. These are the reasons that python, with numpy, scipy, and the huge quantity of other libraries available tend to draw scientific use.
wrt to type signatures, many of those ancient FORTRAN libraries are written with implicit interfaces, so bugs are likely to show up. I came to learn this when compared some versions floating around with the patched versions supplied with scipy.
My aim is not to bash, but justify scipy is a solid piece of software, based on known developments, not just a shiny "new" thing.
I think the foundational libraries of the scientific Python ecosystem are definitely well taken care of. I think a lot of that care comes from “brute forcing” the ecosystem to make it work, eg distributing native compiled C/Fortran code seamlessly on a bunch of platforms with entirely new (at the time) package managers like conda, or wholly new scientific distributions of Python. My observations are more to do what’s built atop them.
If you're looking for a way out of python because of specific performance issues you face I'm not sure why you did not just go with Julia (unless you really wanted to do Common Lisp).
Try pyinstaller if you need to compile python into a self-contained binary.
So there’s definitely a valid maintenance point of code needs to continue to be extended (though I’ve had no trouble finding or training Common Lisp programmers), but Common Lisp code simply doesn’t stop working on the timespan of years.
Any hard-earned advice on how best to work around this in a scientific context without wasting too much time? That is, which areas of numerics / plotting / stats have a useful Lisp library vs. when it's better to write your own? There are a lot of apparently-abandoned numeric libraries on github. Do you have to learn LAPACK & SIMD/AVX directly to be productive?
For context, I escaped to Python from Matlab 7 years ago, but have grown to share many of your opinions on Python. Looking for where to jump next. The short feedback loop of Lisp (condition system / restarts) is appealing.
Secondary to that:
- Learn to use FFI very well try hard to find libraries written in C.
- Familiarize yourself with the structure of LAPACK and what it offers.
- Learn to use a profiler and debugger (if using Lisp: SB-SPROF, TIME, SLIME, and SLDB).
- (if using Lisp) Contribute useful things back to existing libraries, like MAGICL [0].
In my opinion, Lisp has no good libraries for plotting. I always have to plot by using another tool.
SIMD/AVX are things you use directly in SBCL if you want to achieve very high FLOPS.
Maybe it’s not the best analogy, but scientific programming in Lisp is currently like woodworking (compared to building IKEA with Python).
This isn't a criticism of your code, it just reflects a larger issue in scientific programming: most code is never read, edited, or re-run by anyone other than one sole author. If that is the status quo, most considerations about languages don't even matter anymore, since the biggest issue is normally just finding a good balance between functionality and ability to collaborate, and scientific culture has lopped off the collaboration side of the equation.
This choice may also feed into that culture, a positive feedback loop of cloistering: if I were in your field and noticed you had a code base, I would have to go out of my way to run your code or improve it, so I probably just wouldn't.
Seriously. Use poetry to manage your venv + deps. It solves 99% of the common problems I find about conda, venv, pip, etc.
My understanding and limited experience is that US universities focus more on software engineering and theoretical computer science, with numerical methods (even very basic ones) left as a later elective/graduate course.
That said, do you guys really learn/teach the shooting method, Lagrange interpolation, predictor-corrector methods, QR factorization, etc. as 1st/2nd-year undergrads? For what majors? I feel like this would come at the cost of deferring other topics to later years.
Yes, we also covered other stuff like splines, finite difference method, too. I think Finite Element was third year where I last taught.
"Major" refers to the same things you mentioned (physics, math, EE, etc.), it's not any different in the US.
> Yes, we also covered other stuff like splines, finite difference method, too. I think Finite Element was third year where I last taught.
Sorry to repeat the question, but again -- in what major (or "for what degree" in your terminology) did you study this? (CS? MechE? Physics? EE?)
In that department, beyond the courses covering this stuff which were compulsory, there were additional optional courses for 3rd and 4th year students to do C programming and more advanced numerical methods and stuff like programming paradigms for HPC.
And I'm not saying this is a good or a bad thing or trying to justify it, just explaining how it is and what some of their rationales are.
Not sure if you'll see my reply, but in the US this is generally not true for EE. If you're CompE, perhaps. In my EE undergrad, you were required to take only one CS course. It was otherwise heavier on math and circuits/electronics. By the end of the 2nd year, you'd have had two circuits courses, and have taken ODEs, and 3 semesters of calculus. Even the junior year is focused on electronics, E&M and motors - although you can start taking other electives (digital signal processing, etc).
But numerical methods was not required for us. It was a senior level elective.
It is, if the UK doesn't have minors but only majors (e.g. you determine from the start what you'll be studing).
Sure. But the question is "can you do the inverse in the UK?". That would be the difference then -- not having the ability to "play around" before settling.
There are no majors in the UK, and no equivalents of the major/minor dichotomy.
There is just a straight degree in X, studying topics about X. You don't get to decide your major later on, and you don't get to throw in a minor to your degree as well. You study a specific thing from the first year.
The focus was on the mathematics and concepts. We were not taught how to approach these problems using computer code. We were taught very little computer programming in general.
When we were taught how to solve problems using computer assistance the focus tended to be on using specific software packages and not on how to write code.
I.e. using ANSYS for Fluid Dynamics problems or using Excel's "Solver" for simultaneous linear equations.
Edit: QR Factorisation was covered in my first year Linear Alegbra course. For Matrix stuff we used software called "Maple" which was installed in computer lab at university I have never heard of it again since I left uni so I don't think it is all that common. We mostly had to solve problems by hand on paper anyway - which I hated.
It's certainly possible to take it in your second year where I am, but usually students opt for other classes that will help them get an internship before they go to numerical methods, so it's a common third year class IMO. It doesn't normally get pushed off farther than that though, since it's a common prerequisite for fourth year classes.
In general, this means that the first two years of all physics/eng degrees in the UK cover almost identical material across all Universities. It's only after that you start to get any choice, and those choices usually relate somewhat to the expertise of the academics at that institution - I did my first degree in a University with very strong condensed matter theory group for e.g. so there were lots of courses on that.
I work at a place that has a mid sized engineering department. The people who regularly handle math related programming tasks tend to have degrees in the physical sciences, and coincidentally are older than 50.
So there was a huge need to have a survey class like this to get everyone to a common baseline level of knowledge. Some knew math but not programming, others knew programming but not math, and still others knew a little programming and a little math, but not quite enough of either.
Of course, many students do study calculus in high school and can sometimes skip these courses in university, but the curriculum assumes they haven't.
Here everyone doing a Physics/Maths/Engineering degree will have A Level Mathematics qualification and a Physics A Level qualification. In the pure Maths part, it covers integration and differentiation (inc. of trigonometric functions, log, exp and by parts), vectors, and 1st order ODEs as a minimum. There is also mechanics though depending on the school you study at, this did not used to be compulsory (you could choose to do Statistics instead).
In Physics A Level, it covers general physics topics - using SUVAT equations for e.g., electric circuits, radiation, some very basic QM like photoelectric effect. Knowing calculus is not a requirement for that qualification as it's designed to be independent from Mathematics A Level (at one point you had to take it as well to study Physics, but declinining numbers of students caused them to stop that), but to study most STEM subjects at University you do need both qualifications.
Around 50% going on to study for one of these degree courses will go beyond this and will have studied for an A Level in Further Maths. In this you do second order ODEs, complex numbers, matrices, conic sections, and more mechanics. In the Universities I've taught/studied at, the entirety of these Further Maths topics is usually covered in a 1st year + 1st semester course before going on to do harder stuff beyond that.
So from that mathematical background, everything numerical methods-like assumes you already have that knowledge almost from the beginning at University.
I think it's one of the more interesting algorithms in general.
Is there a Github link to notebooks?
I've been teaching Python to adult non-programmers for a few years now.
One issue is that of order on what to teach.
What do you teach after you teach variables and basic data types?
There is no one perfect solution.
This particular book teaches dictionaries, lists, sets (and NumPy arrays!) before branching and iteration.
Good news - the examples in this book do not use L or list as variable names, which even some seemingly respectable books do.
Personally I feel that branching and iteration (over strings) should come first before progressing to more complex data types.
If you learn of lists before you know of iteration you can't really do much with lists.
Note that in practice you can teach part II (numerical methods) without 90% of part I (python programming) given that your audience has some minimal programming experience (not necessarily with python).
https://blog.wolfram.com/2019/05/16/announcing-the-wolfram-c... https://reference.wolfram.com/language/WolframClientForPytho...
It is also possible to run the Wolfram Kernel inside a Jupyter Notebook: https://github.com/WolframResearch/WolframLanguageForJupyter
Btw, not sure how to get the label work.
tl;dr: I bought a copy
oooh just checked and NAG do python library's now as well so that might be teh best solution.
Usually is like this. Customer happens to stumble some easy 3 lines program that solves the problem in an easy subset and after that it will create a full program around that one with all the blings. Then goes into production after a month and 15k USD lighter and its users are using it for full scale and the solving time now is in hours. Then it will search for an expert to solve the impossible problem of speeding up Python and curses at all the Indians that are scamming him and all experts who ask for another 15k USD to actually implement the program correctly.
I shit you not, last one was a data scientist who paid like $3/h some Indian dude that got the job done in 2 days, took the money and contract was closed under those terms. Then this data scientist was all over his job posting crying that, while the initial matrix was in dozens of columns/rows now the same Python program would take 3 days when throwing at it a matrix that was having millions of columns/rows in size. I mean, it's several orders of magnitude higher, I was amazed that the program actually finished the job after 3 days instead of just crashing.
So I had to drill into his thick skull the idea that while he initially went to war against Lichtenstein and won, definitely cannot win against US army using the same weapon. Only after that he agreed to scrap the Python program altogether and finally go with C++, because speed does matter in the end.
Like I said, I love Python, especially for anything math related.
Or, more precisely, you can easily wrap your algorithm written in C by a bit of boilerplate Python code (if that is your fancy).
edit: grammar
I could respond to it but there is not much to be said. I think you have made my point.
Honest question, when you read the line "[...]Chinese that are good at math [...]" do you read that all Chinese people are math-wizards? Because that is not what it says.
What about "[...] black people that are fast runners [...]"?
all 3 examples, including the one that turned on your torch of virtue, describes a sub-set of a group, the primary attribute of said group and nothing more.
If anything the implication that the guy you were replied to is somehow biased and "racist" against the absolute-plague-tier of disproportional scammers coming out of India is based on nothing but your inability to differentiate between "broadspectrum-racism" and "critism of a subset of a group"
A closer analogy will be: "He was lost in New York City. Later, he cursed at all the Blacks who robbed him." Or "He had an intense negotiation with the financiers. He later cursed at all the Jews who were scamming him."
As you may note, the term "jews" or "blacks" or "Indians" (in the original comment) is not merely stated as an adjective to describe the individuals, rather it is used in pejorative sense to denote a cultural trait within the group that makes them act in a particular manner. A child comment by the original poster makes his prejudice quite clear: "Probably because Indians are the ones that are leaders in scamming? "
I get your whole point about talking about individual, subset, and group, but it looks like just a defence for calling Indians "world leaders in scamming.", rather than some data based, dispassionate description of the situation.
Edit: grammar
I'm making extreme examples out of the sentence, but putting something 'awesome' with it. Being good at math / Fast runners etc - to make the point very concise and on point.
Had i run with the theme and went "White people who shoots up schools [...]" or "Black people who sell crack cocaine" you would have likely missed the point entirely because I'm using negative-stereotypes.
That the child-comment elaborates his thoughts into racist ramblings is frankly irrelevant to me. The guy is clearly both illiterate, insensitive and likely in the silly end of the bell curve.
It should not be. We need to call these people out and put a stop to such behaviour.
Coming to your comments; "positive stereotypes" just earn a gentle laugh while "negative stereotypes" lead to racist behaviour with a disproportionate impact on the real world. They are not the same.
Yes, because whether he is a racist or not is not relevant to the point i was making.
> We [...]
We, do not need to do anything. But go ahead, engage him, feed him with the social interaction the rest of us deprive him of - because he is a shitty person.
Waste your time all you want.
> Coming to your comments; "positive stereotypes" just earn a gentle laugh while "negative stereotypes" lead to racist behaviour with a disproportionate impact on the real world. They are not the same.
Predictable that you are missing the point entirely. Please reread the exchange, one line at a time. Else you might think and label me as a racist because you have put up a communication barrier and absolutely refuse to understand my original point.
>the absolute-plague-tier of disproportional scammers coming out of India
Dressing up your words cannot hide the insinuations.
Surely this is an exaggeration?