Why I'm still using Python
mostlypython.substack.com
mostlypython.substack.com
Some of my reasons for choosing Python in the first place have been mooted. Python is free, and multi-platform, but so is everything else nowadays. I think that battle is over.
For my use, packaging has not been a big obstacle, maybe because I tend to use a relatively small handful of established packages. I don't remember needing a package that I couldn't install with pip.
Easy to learn? Yes and no. I think a beginner can ease their way into Python and get productive, if they start with basic scripting and progressively adding a few language features. On the other hand, some features are simply over the heads of beginners, making the code found in packages practically unreadable. But I've never found any language to be better in this regard.
Fast? Sure, with numpy in its domain of use. Or writing your own C libraries. However, I think a beginner using numpy will run into roadblocks that require help from an experienced programmer, who can visualize what the array operations are doing under the hood. So, writing fast code isn't easier to learn in Python than in any other language.
The editor should have a final option, that turns your code brown if it's just so bad that nobody should ever have to read it, including yourself.
Like BASIC back in the day it's a language that allows newcomers to do things long before they are comfortable with the added syntax requirements of C based languages
Despite that the sky is the limit, almost all problems a developer will encounter can be tackled with python. Yes for many problems there are languages that are better suited for the task, but in most cases a project does never even reach a state where that will matter.
And it's installed by default in many Linux distributions.
This is especially true if they've already tried setting things up, and have made a mess of it. Because WinPython is self contained, it can work on a computer that already has a working or non-working Python installation.
A big pain point still remains when it comes to packaging and distribution.
It's not insurmountable, but if you're used to compiling and/or cross-compiling and shipping binaries, shipping Python apps can be headache inducing. The headaches increase when you're working with Windows or macOS, or using bleeding edge Python versions.
That said, for 99% of Python apps out there, the default tooling should be absolutely fine for packaging and distribution, as should 3rd party solutions like PyInstaller or Nuitka.
It may depend on what types of libs you're looking for, of course. It's definitely missing a lot of stuff.
Now people talk about Python's standard library being "where code goes to die," they want to remove old stuff, makes me sad.
Not because the Python maintainers are intentionally doing a bad job, it's just that their mission isn't to substitute for your research. To them, "Does last year's Python script still work?" is always higher priority than "What's the best way to do it today?".
And what Python's security team decide constitutes a reasonable security fix may conflict pretty badly with what you need.
I would assume the Python devs review the code for a package before including it in the standard lib, right? That seems like a pretty big deal, well beyond someone just telling you that they tried a package and it worked for them.
It doesn't make as much sense for Python's standard library where you'd need to be paying attention to goings-on in the community to know if one of the batteries that are included has been judged expired and will be removed.
Give me a batteries included standard library please - much easier to audit and track.
If only Mathematica was free, I think it would become the defacto language for scientists.
The most important feature of Mathematica is that it is functional and not object-oriented. Mathematics is functional, not object-oriented. So thinking in Mathematica is as easy as thinking in maths. Thinking in python is to force yourself to think in an unnatural way.
There is a lot to it, and in my experience it's a very long road to getting your first problem solved.
There are a lot of very cool pieces but I want it to be something I can pull off the shelf once in a while and be productive rather than something that I need to be an expert in before I can get started.
Of course. It’s one of the few languages that I think are worth it.
I’ve been waiting for Wolfram to open source it for 10 years as I can’t afford it, or really justify why my org should use it.
Wolframscript is free (not guaranteed to stay that way, as it isn't Open Source, but free for now).
It can AFAIK do pretty much all Mathematica can.
Only part missing is the Mathematica IDE/GUI/Notebook which I personally dislike profoundly anyways.
Best of all, since it's a batch tool, it can be integrated in a Makefile and/or data processing pipeline without that nasty GUI showing up at all.
Python has object-oriented features but there is nothing that requires you to use them; you certainly don't have to express every Python program in terms of classes and methods, the way you do in Java. Doing functional programming in Python is common. Is there a particular aspect of doing functional programming in Python that you find to be a roadblock?
On the other hand, all the code I wrote in Mathematica during my PhD was functional, because it is just easier to write functional in Mathematica.
I'm not sure I see why. The object orientation in libraries like numpy or scipy is to provide the user with fast implementations of the kinds of mathematical objects they will need to do computations. But unless you are defining additional such objects in your own code, there's no reason why your own code needs to be object oriented. You can just write ordinary functional programs that use the mathematical objects the library provides.
For example, say you are doing matrix computations. You are of course going to use the matrix objects provided by the library; but your own code shouldn't need to define any new matrix objects. Your own code can just be straightforward computations using the existing matrix objects in the library.
I beg to differ. If by "math" you mean arithmetic and calculus, then sure. Combinatorics, graph theory; probably most of discrete math is very much object-oriented.
In python, in popular libraries, the above will be attained by something like
G.shortest_path(a,b)
which seems to imply as per OO that graph G has a property shortest_path. In mathematics we don't think like this. Because it prevents mathematical abstraction. An abstract algorithm, like say Eigenvalues can be applied to a matrix M or to a graph G. In a functional language, the user would just do Eigenvalue(M) or Eigenvalue(G) as needed.
In python, these would be M.eigenvalues() and G.eigenvalues(), which makes them distinct.
I wish all the others did as well.
There are also functions that operate on graphs, because the alternative would be to stuff perhaps thousands of methods into the graph class -- which would be extremely annoying to maintain as well as abysmally slow.
Funcional programming is used in Coq because it captures logical chains, for theorem PROOFS, but not for specific COMPUTATIONS, such as eigenvalues.
Mathematics (outside of stuff like category theory etc) is simply sets and mappings between sets. Anything more complicated is forcing the user to think in a way that mathematics does not.
Hi, mathematician here. Please read the source of SageMath: written by mathematicians, for mathematicians. Therein, you will find thousands of classes. Almost all computer algebra systems, written by mathematicians for mathematicians, have a notion of classes.
> Mathematics (outside of stuff like category theory etc) is simply sets and mappings between sets.
This is an extremely narrow-minded view of mathematics. Mathematicians thrive on abstraction, (category theory is literally mathematics; that's a very strange exception for you to carve out), and if we were to boil everything down to "simply sets and mappings between sets" then we'd be bogged down in utter tedium and nothing would ever get done. Object-oriented programming, specifically class hierarchies and inheritance, are extremely valuable for doing all sorts of math at high levels of abstraction.
Hell. If I were to be especially pedantic, I'd point out that category theory itself is "simply sets and mappings between sets" except that sets aren't quite large enough so it's actually "categories and mapping between categories".
...what?
On the other hand, the non-mathematical functionality of Python of course is much better developed!
Second, unless an organization is enlightened enough to approve a site license, specialized software tends to create a "silo" of people who can use it, and forces it to be used in a centralized fashion. With Python, I can literally install my tools on every computer that I touch -- in the office, multiple labs, at home, etc. It makes it more likely that a tool will be woven into "how I think."
I can also share things and/or encourage others to try them out with minimal friction. Almost as easy as sharing an Excel spreadsheet within an organization that has purchased a site license. Free means free site license.
The friction of approving a license also discourages people from learning a tool by just giving it a try on some trivial project, or taking it home.
Similar problem with Julia and Fortran even. They all have their niches and are great at certain problems, but Python (and C/C++) rule the overall landscape. in HEP, for example, it’s even the case that a full batteries-included C++ environment (ROOT) has slowly given way to python.
English is not really a beautiful language - it’s a wild mess of different influences, but it’s easy to get to a basic level of fluency as there are few rules (but many exceptions due to the many influences). And it’s in use by large parts of the world.
On the other hand, Latin is beautiful and pure. It has more rules, but very few exceptions. You can also see how it influenced other languages (hello Jupyter notebooks…) But it’s simply not a very practical language today, as there are few people who use it.
... That we know about. It's a dead language, so the exceptions would've been lost to time as it hasn't been spoken in ages.
Your point stands regardless, just felt like being nitpicky
Either you take the time and trouble to understand what you're actually doing or you're just a lazy wizard with some facility for incantations.
I don't think there's even a (reasonable/non-contrived) way that a package could be available to you that pip wouldn't install?
(Or do you mean that you don't remember desiring a library functionality that wasn't available in some package?)
Also there are probably tons of "packages" which were never published officially to pypi but exist on github, that people have made public but decided not to maintain.
I'm doing this from memory, it's hardly a frequent thing for me, but err something like:
pip install -e git+https://github.com/vonseel/...Both might require you to have compiler toolchains installed.
A practical example: on Windows, try 'pip install tokenizers'.
py -3 -m pip install tokenizers
...
Building wheel for tokenizers (pyproject.toml) ... done
Created wheel for tokenizers: filename=tokenizers-0.13.2-cp311-cp311-win_amd64.whl
1. docstrings
2. keyword arguments
Docstrings mean I can do REPL experiments without the additional friction of opening a browser to RTFM. Keyword arguments reduce how much I have to lookup the semantics of a function's signature.
Getting the editor to measure the cyclometric complexity of the code and then colour offending blocks?
That's too bad. I taught myself Python last year but have kept it pretty "101". That is, I write code simply. Since I am new to Python I don't know if the unreadable code you're describing is because it needs to be (perhaps I am still tooling around in the parking lot of Python) or if it is the result of programmers being too cool for school (ha ha).
Perhaps if the latter I ought to blame the language for giving programmers a basement of esoterica to plunder rather than the programmers themselves.
As obtuse as C could get with pointer arithmetic and the cryptic looking "question-mark-operator", there really wasn't much esoteric stuff in C. Contrast that to my experience with Swift where seemingly every week or so I would come across cryptic code where someone was using a capital T or some bizarre case expression I am unable to mentally parse.
> Turbo Pascal
You suddenly made me miss an ex-girlfriend. Maybe they still code in Turbo Pascal in Portlandia.
I can dream.
After that... C++ was fun initially and later felt way too featured. I don't like C++ code now (especially with heavy template use and stl and boost).
I now code in Python (because most of the things I work on are in Python) though I don't like the dynamic typing.
My potential new love is Rust but we are still working out some issues
In what ways do think it's painful?
The other thing which rubbed me the wrong way was that the python was happy to run the code with completely wrong type hints.
I guess I went into it with wrong expectations, even though it says right in the name - it's "type hints". The whole experience felt more like a formalized documentation with partially working optional verification (which can't really be relied upon).
They are also for static checking.
If you choose to run the code despite that failing (which you can, because there is no execution dependency on type-checking) that’s a choice, but its kind of odd to complain that Python lets you do that.
But do write type hints. I recently got thrown into a large-ish project where neither types nor docs where used. Trying to figure out wth a parameter was supposed to be wasn't a pleasant experience for a newcomer. In addition to improving DX, I also believe it's alot more effective in the long run.
I saw how these guys were developing: write code, run code, deal with the runtime crashes they encounter, then run code some more and deal with other unexpected runtime crashes. It would have been a lot faster and more stable if they'd just used type hints and static type checking, as their IDE could've easily found many of these bugs for them immediately.
Mypy's type system is quite advanced compared to some statically typed language like C and Go; it has generics, union types, inference, and it can prove properties of types from control flow. I work mostly in Rust, Haskell, and Python, and I rarely find Mypy limiting.
You do have to embrace types though; if you have a dictly typed program where everything is a Dict[str, Any], then putting that in annotations isn’t going to be very helpful; converting the dicts to named tuples or dataclasses is.
Partly that's because the ecosystem support isn't there - lots of libraries don't have types, or have types that behave oddly, or even require their own plugin for mypy. I suspect that's going to slowly change over time, but I feel like the infrastructure doesn't feel as strong as it did in the early days of Typescript, and the gradual change feels even more gradual.
Partly it's just that the syntax is painful as soon as you want to do anything remotely complex. For example, generics are defined in different ways depending on what you're making generic, and the TypeVar system is usable, but ugly and very unintuitive. There's also missing syntax for things like inline type assertions, which aren't great, but are often useful for mixing static and dynamic code together.
And I think partly it's also that Python has a much higher level of dynamism in the type system - lots of "clever" things are possible in Python that just wouldn't be possible in the same way in Javascript - which means that mypy has a much more difficult time in trying to describe the idiomatic Python patterns that people are actually using. In Typescript, figuring out the correct types feels like an extension of writing natural Javascript code and understanding the flow of data through the program. With mypy, it feels like I'm restricted to writing the subset of Java-like code that conforms to the type checker.
It's a disappointing feeling, because I like static types, and I had hoped it would help solve the problem of large codebases in Python. But in my last project it became so painful to use, and felt like it was adding so little (and preventing so few actual issues), that I kind of regretted pushing for it.
I hope a lot of these problems are just teething issues, and that over time it'll get better. But right now I would be very cautious about using it in production.
JS has Object.setPrototypeOf(), among other things, so I doubt this.
Or even just TypedDict, which often works without any changes besides annotations.
> I've used Mypy since 2016 on big as well as small codebases, and it has been extremely useful for me.
You can both be right. It's both not very good compared to a strict type system, and still way better than not having it.
People do have bad habits of cramming everything into dictionaries but if you do some type hinting and use data classes heavily you’ll really have a good experience.
I could take or leave mypy personally.
Having a Scala code base not compile because someone went all-in on type-level programming, and now simple changes require an understanding of category theory and the associated compiler flags .. that's real pain.
python's type-documentations are useful but sometimes they are just wrong which makes it impossible to actually trust in them.
> Having a Scala code base not compile because someone went all-in on type-level programming, and now simple changes require an understanding of category theory and the associated compiler flags .. that's real pain.
Well, the same can happen in python if someone goes crazy and uses a lot of reflection / dynamic changes (i.e. overwriting builtin methods etc.). In both cases it sucks and should have been prevented by other people, but at least in the case with Scala you still have a compiler that can help you "unroll" those changes because it tells you when you screw up. In python you can only have tests or pray.
I wish I'd kept it but a few years ago someone did an analysis of all the public python code in github to see what functions were called the most.
#1 was `type()`. <shocked face>
Moreover, it has static type checkers with varying degrees of support for type inference.
And static types are for people who are too lazy to write test code.
Where did you come up with this?
Functional tests will test the correct types, too.
Edit: If you need an object that swims and quacks, you don't need to care if it is a rubber duck or an animal.
Static types are very useful for compilers but looking at a function and seeing int -> int -> int -> string -> bool -> int, says very little about the semantics of a program. It's always the names and documentations that tell human beings how to make sense of a program.
When we put things in record types, the sense-making value isn't in the static analysis but in the fact that our vague collection of parameters now has a proper name that indicates what it's all about.
Sure, but on the other hand a type of APIProxyEvent -> LambaContext -> APIGatewayProxyResponse says quite a bit more about the semantics.
Unless a function is highly abstract, int -> int -> int -> string -> bool -> int is probably an underspecific type signature.
EDIT: To be clear, I generally find that the thesis “you don’t want types, you want better names” comes from assuming bad types, and suggesting replacing them with good names. And, for casual inspection, yes, good names may be superior to bad types. On the other hand, I can’t statically check good names, I can statically check types (good types, or bad-because-underspecific types, but good types, as well as telling me more as a reader, will also statically catch more possible errors.) Ultimately, what I want is good types and good names,
When you're using types as a means to check names you're likely to misuse types. Synonyms are a good example of this, where people will make so many types each type only ever occurs once, instead of having a well named variable of a more generic type.
Different but overlapping needs, yes.
> For example, if you have types defined in your program, you could literally replace them with some random characters and the type-checking would be equally good, for the compiler, even if you don't undestand a thing
Sure, but for human consumption you want good names of types, not just good logic of types, just like you want good names of variables.
But, while you can (without types) overload names of variables with the human information that would be in names of types, this is bith less ergonomic that separating the two kinds of names, and doesn’t support type logic the way types that are both well-structured and well-named do.
> The value of types really is in the structure they represent, and enforcing certain constraints, not in any human understanding of the program.
I could not agree more strongly with this, it is no more true of types than it is of the rest of code: yes, with any part of code the logic is was matters functionally, but humans maintain the code, so if its not written (both names and structure, within the variations that produce correct behavior) for human understanding, its less useful, and potentially useless.
> When you're using types as a means to check names you're likely to misuse types.
I don’t know what this is referring to. If there is a statically verifiable feature, it is a real type constraint.
> When you're using types as a means to check names you're likely to misuse types. Synonyms are a good example of this, where people will make so many types each type only ever occurs once, instead of having a well named variable of a more generic type.
Synonyms/aliases (at least in languages I am familiar with) are explicitly not types, but alternate names for types. They aren’t checkable, only the underlying type is. They are useful in much the same way as named constants.
Take for example, the numpy.dot function. This is from the documentation:
> numpy.dot(a, b, out=None)#
> If both a and b are 1-D arrays, it is inner product of vectors (without complex conjugation).
> If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a @ b is preferred.
> If either a or b is 0-D (scalar), it is equivalent to multiply and using numpy.multiply(a, b) or a * b is preferred.
> If a is an N-D array and b is a 1-D array, it is a sum product over the last axis of a and b.
> If a is an N-D array and b is an M-D array (where M>=2), it is a sum product over the last axis of a and the second-to-last axis of b:
So to figure out what this call to dot actually does and what it's output type is I need to know the actual types of 2 inputs. Of course, those inputs are the results of other function calls, that are also dependent on the types of multiple inputs. When reading code you have to manually keep track of each type as you read through it, you have to look up every function to see what it does and what it can return under what conditions.
Programming languages are not for computers, they are there for humans to understand what is going on. Python absolutely fails at this because python code is almost totally unreadable. It is at best a write-only programming language.
Sure, agreed, but if a program contains poorly named types then it's a good bet that if types were not mandatory, those authors who could not be bothered with properly naming their types would be equally incompetent at naming their variables, parameters and functions.
IOW, if author competence is important for typed programming to be readable, then it is even more important for untyped languages!
Strongly typed languages is simply a trade off where you get peace of mind by paying for it with extra time and effort spent. For some people that’s a no-brainer and for some people that’s just something that gets in the way.
My background is mostly in sysops and monitoring, and for me types are a life-saver because I value stability and predictability over almost anything else. If my job was something more similar to “ship new stuff fast” I’m not sure that would be the case.
A couple of comments here:
1. Python is very strongly typed, every single item has a clearly defined type. That said, it is dynamically typed, meaning that the variables are not a "containers" for predefined types, but simply labels on objects.
2. What brings benefits is not static typing itself, but static analysis which is mandatory in most statically typed languages (i.e. the program won't compile/interpret if the analysis breaks). With tools like mypy Python has a perfectly capable static analysis - even without any type annotations, although they massively improve it - it's just not mandatory and the program will happily run even if it fails.
Ironically, the slowest moving codebase I have ever seen was written in Python. It was impossible to "ship new stuff fast" without breaking things. I think Python works up to a certain size/complexity level, then it completely falls apart.
Which is not what P said. They said those that have it tend to favor strong typing; those that don't tend to not.
IME I've seen the same thing. It's not a rule, it's a generalization that is often wrong, but more often right. Again, IME. YEMV.
[1]: https://wemake-python-styleguide.readthedocs.io/en/latest/
I could imagine that if developers choose to use less complex objects and less indirection then types in such large projects would be more useful in explaining the data. It just hasn't been my experience so far.
In both C and python I have also see something akin to a mini language inside the large projects, where understanding the code becomes almost impossible without documentation. In C, both macros and pointers to structs with more pointers can do a lot of heavy lifting to hide every detail of how something is being done and just leave the intention. Great if one want to see a high level concept and creating new features, but terrible if you want to know where that one bit of information is being stored and how to inject your own code into the core parts of the project. Similar, large projects in python tend to use meta, monkey patching and other dark magic patterns to really hide the low level details while making it easy to create new features.
With Python I never know, since something might have dynamically added or changed some methods or fields along the way. I almost always end up sprinkling dir() all over just to figure out what exactly is going on.
It has typescript built in so I can very quickly make a script.ts file anywhere and run it in the CLI with `deno run script.ts`. Works flawlessly and I get access to TS's amazing type system without any build step or setup or even any additional files
for any data-intensive scripts I still sorely miss pandas though
You can import synchronously, asynchronously, as a stream, as a string, etc. All the functionality you'd expect is built-in with the Deno module (e.g. `Deno.readTextFile('./my/path.json')`) or even with module imports
On the other hand every business comes up with its own little world.. and suddenly you're in the dark. Here the need for strong and static typing helps.
I've written a lot of Python, taught classes in it, deployed production code... and I still feel a semi-conscious urge to reach for something else whenever I contemplate starting a new project. Something about its approach, syntax, common idioms, always feels just a tad clunkier to me than I'd prefer. The whitespace makes it hard to paste lines into my repl, and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks).
I like dynamic languages, I really like the advances in tooling with type checking and LSP support. I like dynamic notebooks (either inline in VS Code #%%, or straight Jupyter), the massive package ecosystem (but obviously hate the actual packaging tools), cool tools like Rich... but it just doesn't make me as happy to use as other languages.
I'm still trying to articulate why. Maybe I was just ruined by learning Ruby first.
Then rewrite it later when you need to scale.
Recently I chose to rewrite several thousands of lines of Python in Go, because we needed more speed and improved concurrency. Already having a working program and tests in Python was great. After figuring out a few Go-isms, it was a quick couple of days to port it all for big improvements.
Any unit tests got rewritten along with the test's relevant code. Now that we are sticking with this Go based solution, adding more unit and regression feels more reasonable now.
What Python gives you, and what's not easy to get elsewhere, is access to really high performance numerical code, wrapped into quite ergonomic interfaces: Numpy, Scipy, Pytorch, etc. Many other things choose to make Python the glue language, like above, or to embed it (from Gimp to Blender).
This makes Python stand out, despite its shortcomings (which are well-known), and make it an even more reasonable interface language for whatever next big thing. Network effect in action.
I'm confused by this statement. As I understand it, list comprehensions exist to transform data. Inline functions in JS would be equal to lambdas in python, I.e. anonymous functions. So what differences between JS and python do you think is missing?
Lambdas are incredibly limited and encourage the declaration of a whole separate function that now lives physically far from its use.
List comprehensions are also really underpowered and I personally hate the map and filter functions and their goofy list-like types.
Check out linq in .net, streams in Java, array in JS, collections in scala to see the expressive power of a fluent interface on collections
Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time.
> I personally hate the map and filter functions and their goofy list-like types.
You mean iterators? There’s nothing else they could possibly return. Map and filter are lazy and work with arbitrary iterables even ones that never end or have no possible way to define bind.
I think a lot of people have no idea nested functions exist in python...
And since you mentioned dynamic languages and Ruby, I think it might be a really good match for you!
Python has it's faults. Most are handled by a good editor. I only switch from Python when I need something lower level. With Python a company gains:
1. Simple language with relatively low ramp-up time
2. Reasonably fast with proper tuning. I've been on projects processing impressive numbers of RPS built entirely in Python.
3. Comprehensions. You claim they are not sufficient. With them I rarely need to lambda anything. When I do I have `functools` to do any other work.
4. A massive ecosystem of libraries, resources, etc.
A company loses:
1. Projects become difficult to understand at a certain size. This is not properly addressed by even the proper use of MyPy.
2. Somewhat more involved testing process compared to it's contemporary - Ruby.
To compare apples-to-apples you can even look at the interpreters themselves. Python's mainline interpreter has been able to run CIRCLES around Ruby's for years now. I can understand using something like Go for systems level work, but there are few languages that give you so much for very little investment. Even dropbox was run entirely on Python for LONG time before it became too unruly. That's INCREDIBLE given the simplicity of the language.
What you use on your personal time is irrelevant. Your complaints apparently are not experienced by the swathes of companies switching from almost every language under the sun to Python. It's here to stay. At least until Go's ecosystem catches up to Python's. My belief is Go will be the only language to unseat Python at this point. Python will still be around as a "perl-replacement" for basically ever as far as I am concerned.
* I’ve been using it for a long time
* I have lots of friends that use it
* Even though I’m interested in other languages I don’t have time to learn them
These are all perfectly valid reasons! But, they say very little about Python.
* It's preinstalled on my favorite OS
But seriously. I like how fast I can start getting results with it. Sometimes I don't even need to create a file, I simply do "python3 -c ..." or use it as an interactive shell
> Most criticisms I see leveled at Python are still completely unfounded. Many times the criticism can be addressed by using the language in a different way.
Is that true? Functional programming missing, slow, poor module system and REPL, weird scoping, environment and distribution issues, the nightmare that was 2.x to 3.x, tacking on features a la C++, etc.
> So be it; if I’m not working in one of those areas, then Python is still probably the best fit for me. I used to hear that Python wasn’t the best at any one thing, but it was second best at most things. I agreed with that line of reasoning for a long time, but these days Python is as good as any of its peers for many things, and it’s still quite effective in many areas where it might not objectively be the “best” fit.
This is just weird, especially if one doesn't know or use other modern languages. F#, Rust, Elixir, Ruby, Racket, OCaml, etc.
In my opinion, unless Python is being used for scripting, machine learning, or scientific programming, then it's the wrong tool. And I think other languages can even compete with the scripting and scientific programming aspects.
Racket had a moment a few years ago where it looked like it might take off. That ship sailed. It’s the ruby problem with extra steps.
I suspect they mean that Rails gave Ruby its 15 mins. (and why the lucky stiff!) I, personally have no issues with Ruby (I was a very early adopter coming from Perl, C, C++, Python), but find it to be "Python with extra steps" more or less.
But I was mainly listing languages that one should probably look into before claiming Python is a “best fit”.
I also would like to know what the Ruby problem is? Is it referencing meta-programming?
I'm not a fan of the new project, since it means the core team is less focused on Racket than they otherwise would be. But, it's their life.
I've heard people echo the "Python is the second-best language for anything" sentiment a lot. A bit broad and a touch pithy, but over all I'm inclined to agree. It's a language clearly (and explicitly, as said by Guido) designed to be readable, and I think that more than anything else is why I have a case of the warm fuzzies for it. Compare even a small thing, like f strings in Python vs string interpolation in JS -- f strings are simply less noisy and therefore easier to read. Decorators, when used well, make it a fantastic language with which to write libraries that a user can grok quickly. Etc.
If I could design my perfect language, it would be Python, just with clear variable declarations via a keyword such as `var`, optional true static typing and consts, and better tooling -- though, I have to say, after diving in and really learning the ins and outs of venv, poetry, all that stuff, the "Python's tooling is a disaster" meme is IMO a touch overblown; I felt that way as well at one point and in hindsight it was due to me feeling overwhelmed, as opposed to any objective evaluation of the tooling; that said, there is definitely room for improvement.
I myself am incredibly interested in properly learning C++, Rust, Go, and improving my JS skills, but overall it never quite feels like the juice is worth the squeeze, because Python really does just feel like "the second best language for anything." And since my interests are pretty wide -- everything from web dev to scripting to desktop apps to audio (well, Python is perhaps the worst language there), most of the time it just seems more practical to figure out a way to do the thing I want to do in Python, especially because there's usually a library for it already. That said, I'm a solo developer at a non-tech-related-industry job, so I'm not rubbing up against many of the same frictions I see discussed so often here on HN.
As I said, though, I'm still pretty green and am aware that I know just enough to be dangerous, so I'm very open to hear why anyone disagrees with this viewpoint.
I highly recommend taking a look at F#. It is functional first, but it is multi paradigm in that it supports imperative and OOP programming just as well. You can write code in F# effectively how you’d write code in Python, if you’d like. F# is indentation sensitive, has sane scoping, the dotnet CLI tooling, modules, pattern matching, easy and regular async, scripting, notebooks, static typing with type inference (meaning you don’t need to type annotate everything), etc.
I would argue that of everything Python does, F# does it better and more. So if we accept that Python is the second best language at everything, then F# becomes a rather interesting language to consider.
Also, after I wrote the above comment, I checked out Nim, and it in many ways is exactly what I was describing. And Cython as well, even for all its (thankfully continuously reducing) weirdness.
In a different universe its sweet easygoing manner would have created enormous value by enabling all sorts of casual programmers to have some control over this most ubiquitus of computing devices.
Alas we dont live in an era where tech aims to empower users.
NB: There are a number of projects that allow some of this (kivy, beeware) but they need to swim hard against the current.
I like Python but this is probably my biggest gripe.
Apparently it is possible to delete a function.
https://stackoverflow.com/questions/10367414/delete-function
>>> def f(): None
...
>>> del f
My other comment in this thread is relevant too: https://news.ycombinator.com/item?id=34197904
def func(g_func, a, b):
g_func(a, b)
def scope():
c = "c"
def anon(a, b):
print(a, b, c)
print(id(anon))
return func(anon, "a", "b")
def func(g_func, a, b):
g_func(a, b)
def scope():
# NOTE: The `|a, b|: (...)` syntax is made up
c = "c"
return func(
|a, b|: (
print(a, b, c)
),
"a", "b"
)
Note that the `anon` function in `scope` is only compiled once. You can verify this by copying the first version to a file named `temp.py` and then running `python -m dis temp.py`. You can also run `scope()` twice in a row and see that the same object ID is printed both times.
That said, for purely aesthetic reasons, I would like it if Python had some kind of syntax like version 2, but that's tricky with significant whitespace, and I've never found the lack of such syntax to be particularly limiting in Python.
Yes, lambdas are just anonymous functions that are restricted to a single expression, no statements. This is a problem for programming in functional style, because python idiomatically uses statements a lot, and doing a named definition for a single use function is more verbose and less readable in mant cases.
Lambdas with one or more statements instead of a single expression (the actual limitation: “multi-line” is a misnomer) are only an issue because python is a strongly statement-oriented imperative language, and not expression-oriented like most functional and some other languages.
I think if we switch from endless arguments about best languages to the optimal pairs of languages we'd book huge gains (and less online noise :-)
That's not why it has never really been used for mobile development.
For consumer grade applications, the issue is slow startup time due to module imports being dynamic and slow.
For scripting use case by a programmer, tools exist in Python. However use cases are rare, often better served by Raspberry Pi and others.
yes that is still the case e.g with kivy based apps (and apk size too, as you need to ship the entire stack). but such issues could be helped if python was to be promoted to "first class citizen". apparently, at least on the android side, the possibility was considered and rejected early on
in any case, whether python or something else, what is missing is easy programmability on mobile. current devices feel more suffocating than Microsoft Windows desktops ever were. you can also see it in the open source app collections in f-droid which I would say is on the underwhelming side.
who knows, maybe wasm and projects like pyscript will stirr things up
That's just like, your opinion, man
Whenever I think I should switch to faster, more efficient language (C#/Go/Rust/...), I am repeatedly blown away by what Python lets me do, with an incredibly ecosystem of packages, and all the ergonomics of incremental development from a REPL.
Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement.
That's broadly comparable to the time for load the source file from cloud storage.
And that 50ms is processing pages sequentially, using a single core on my little laptop. Plenty of scope to parallelise if I wanted to.
The reason why I'm still using Python, to borrow a slogan from Bruce Eckel long ago, "Python fits [my] brain".
The lack of typing (in my experience you always end up with a bunch of libraries that don't have type hints) means I spend most of my time just figuring out what I'm supposed to pass to a function (I hate the django docs. Give me something like docs.rs where I can quickly scan what functions something implements, and see what they accept and what they return instead of forcing me to read an 800 page novel that STILL doesn't give you that basic info), and writing tests and trying to guard against passing the wrong thing around.
Visual Studio Code's type checker saves so much time and has improved my code quality to no end. It is especially powerful on polymorphic inputs, making sure the different code paths operate on the input type you expect.
I've recently started using Type Hints by default. At the very least, it makes your code more readable. Nothing bad comes from that.
I will say that more than one bug has been found after I disabled the single line hint though! Always surprising to me how powerful types can be.
climb(thing): '''Climb a thing with steps''' # prevent run time crash: if not hasattr(thing, 'getSteps'): hitDeveloper()
Thanks to you and baq!
Any sufficiently detailed python docstring contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a type annotation.
Once you make that big scary change you might have to rerun your unit tests dozens of times before you've found and fixed all of the bugs that your change caused.
A static type checker, plus a bit of foresight and diligence, will give you a much more complete view of the carnage up front, so that you have a good idea about the cost of making that change while it's still cheap to decide differently.
And the best part is that with python you can only bother with the type hints where you need them (e.g. at the boundaries between teams)
And some libraries sould be better documented.
But the compile time error messages of C or C++ or Delphi most of the time do not deliver a shorter feedback loop, because they are generally ugly and hard to understand. And you should write tests in all languages, so static typing isn't a timesaver in that respect, too.
It's gotten a lot better with type hints and typeshed, but I agree, not knowing what type an arg should be and hoping duck typing works out isn't great.
You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement.
Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total?
Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days.
How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?
Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size.
At scale, that could translate into hundreds of thousands of dollars in savings.
But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.
I'm very aware of the business value here! The limiting factor in these types of "messy real-world data" problems is the developer time to get 100+ templates right on all the different variations encountered in the wild. I can iterate on each template extremely effectively in a Jupyter notebook REPL, and immediately rerun a sample of 100 statements for that bank in a few seconds.
While the total corpus of statements I have access to is actually around a million, no one cares how quick processing them all are if the extraction isn't reliable enough!
I have a soft spot for the work you’re doing since I’ve spent a good portion of my life now extracting data from PDFs and can appreciate the joys of the process more than most.
If you had 20 bank accounts sending you monthly statements for 100 years you'd have 24,000 of them. 50ms each would take you 20 minutes to process the 100 year backlog.
If you have a million of them to process then you're a bank or similar type of institution that can devote resources to this, either computational or developer time to optimize things. A 128 core c6a.32xlarge would turn the runtime from 4 hours into 1.9 minutes and cost $5 to run for an hour.
Ah well, 5 tons of CO2 is a fair trade off for being able to use Python, right?
You can solve scaling problems in many different ways with different costs. Any reasonably sized company has a Kubernetes cluster, just launch 10 instances and you're down to 24 minutes instead of 4 hours. You could also buy faster hardware. You might increase single-node performance by rewriting in a performant language (this won't help much if I/O is the bottleneck though, which often is the case). Talking to your data supplier to check if they can send the data in a machine-readable format is probably also something to consider. Or you might conclude that any effort is not worthwhile at all. Who knows?
Scaling problems are not be avoided as a negative thing, they're beautiful and ought to be celebrated. What's not beautiful is letting the thought of potential scaling problems lead you to make decisions that optimize for long-term results instead of short-terms results.
Also, I love the syntax. It reads like a sentence.
What’s that, nowadays? 25 million cycles, 75 million instructions?
Python isn’t plenty fast, modern hardware is.
For use cases where CPU is the bottleneck, such as numeric simulations, Python users reach for something like numpy, where all the CPU intensive stuff is being done in C extensions, not Python bytecode.
Your counterpoint is that it’d be slower if it were running on a Pentium 4. Do you honestly think that they don’t know this?
Yes, this is precisely the point. Things are at a point where Python’s benefits are worth more than what another language would bring in speed improvements. I imagine that there would be less of a benefit to busses if we had to Flintstone them everywhere instead using a combustion engine. That’s not the current reality though.
I think it's fine to like Python as a scripting language, but arguing that Python is fast is simply laughable.
I forsee Python's reputation going the way of PHP's (< 5, not modern), and statements like this will definitely contribute to it.
> I bet I'll build it faster with pure Python.
What? We're talking about runtime speed, not development time...
Anti-up.
Once that's working, the code moves from the Jupyter notebook into a proper code module with unit tests. Then the emphasis is on building the production system rather than manipulating the data.
I get what you mean with unit tests. Just for me it feels further away from the data, which is where most the complexity in my business domain lies.
I'm interested in management of domestic documents and similar institutional correspondence, and I'd love to hear more about how you approached and solved your problems with the PDF bank statements.
I have lots of bank statements ... and other documents like utility bills, rates invoices, etc ... and I'd love to just be able to feed them to a script and automatically tease out salient details.
When you scale, certain modules will start becoming the bottlenecks. At this point one can just re-write the given module as a sidecar and Python by design is easy to bind external libraries.
stuff = [cleanup(s) for s in stuff if s.unprocessed]
Downsides? Dependency managment, tooling, speed
Python comprehensions in particular are unfortunate because the flow of the query, like in SQL, is not linear - the projection part is written first, but happens last. Thus, reading a non-trivial query requires moving back and forth to resolve the references.
Compare that to C# LINQ, which is also a kind of sequence comprehension. That one not only has linear syntax where the data flow in the query is strictly left to right, but it's explicitly defined by mapping it to functional pipeline style. Thus, it ends up being a strict improvement on the latter.
(Another example of "good" sequence comprehensions in this sense is XQuery FLWOR)
Chained pipelines (`foo.filter(...).map(...)`) are fine to me, certainly more verbose then comprehensions for better & worse, but I get lost in pipelines expressed as nested functions (`map(filter(a, ...), ...)`).
If its more than a very simple expression, with one simple for, and no if or very simple if, I try to use at least (more, with parens, if necessary): one line for the result expression, one line for each for clause, and one line for each if clause.
But, while the boundary is subjective, it doesn’t take much before functional pipeline style is clearer (Python lambdas being more limited and syntacticaly cumbersome, functional pipeline style isn’t as clean in Python as it is in other languages, though.)
heads :: [[a]] -> [a]
heads xs = [y | (y:ys) <- xs]
-- heads [[1],[],[2,3]] == [1,2]
heads' = map head . filter (not . null)
It does as of 3.10:
https://peps.python.org/pep-0622/
Not as sophisticated as what Haskell can do, I'm sure, but I think it's a good addition to Python.
(a -> Maybe b) -> [a] -> [b]
heads = mapMaybe safeHead [
if List.isEmpty items then
p “No items”
else
ul
[
for item in items do
li [ item ]
]
]if len(items) == 0: return [p("no items")] else: ul( [ li(item) for item in items ] )
I am also not a fan of convoluted list comprehension (in python) so powerful is not necessarly a good thing.
And consider this example:
[
h1 “items”
if List.isEmpty items then
p “No items”
else
ul
[
for item in items do
li [ item ]
]
]I don’t see how this can be done in Python without using mutation, which makes code scattered across multiple statements
With Python3 there is a move towards "idiomatic" Python code, which to me means moving away from clear and concise to the old programmer trap/readability nightmare of "look how many things I can get it to do with one line of code!"
Not sure if it would make my top-5 Python-hate list (1-4 would probably be dedicated to packaging and distribution), but list comprehensions definitely feel like a misstep that contributes little value to the language.
stuff = stuff.map {|s| cleanup(s) if s.unprocessed }.compact
1. What does the anonymous function return if s.unprocessed is false?
2. If it returns nothing, do I get a missing value element in the iterable that results from map?
3. And what does it mean to compact the result of the map call?
The Python snippet raises none of these questions in me. But it could be because I am familiar with Python’s quirks.
Interestingly I wasn't exactly sure what the python code would do, which is why I used the compact. In the python code, does 'stuff' end up only having the result of the cleanup function for items that are unprocessed? Or will it have all of the items, and the ones where unprocessed returns false just end up being returned unmodified?
My Ruby version returns the processed subset of items that were unprocessed. The python version was not clear to me, since I don't know what happens if s.unprocessed returns false.
So basically, it sounds like we were both confused by the same parts of the language we are unfamiliar with... what happens to elements where s.unprocessed is false? It was clear to me that in Ruby it returns a nil, but I have no idea in Python... what happens?
Python’s list comprehension combines map and filter.
[process(s) for s in stuff if s.unprocessed]
map process (filter isUnprocessed stuff)
I find the syntax of list comprehensions ugly and inscrutable.
This is basically selling point for comprehensions. You can't add complexity to them so any comprehension you see fits a few common loop patterns you can grok immediately.
If you see an actual for loop in python it means "pay attention something odd is probably going on."
This example…:
a = …
vs = [
v(x)
for b in a.f()
for x in b
if x and c1(x)
and not c2(x)
]
a = …
def each_v():
for b in a.f():
for x in b:
if not c1(x): # comment
continue
if c2(x): # comment
continue
if x:
yield v(x)
vs = set(each_v())
// JS
stuff = stuff.filter(s => s.unprocessed).map(cleanUp);
// Java
stuff = stuff.stream().filter(s -> s.unprocessed).map(Main::cleanUp).toList();
// C#
stuff = stuff.Where(s => s.Unprocessed).Select(CleanUp).ToList();
And if you like Python’s dictionary comprehensions:
{str(n): n * 2 for n in range(1, 11)}
Enumerable.Range(start: 1, count: 10).ToDictionary(n => n.ToString(), n => n * 2);Sure, but the Python code is so much nicer.
I don’t even like Python all that much but it’s comprehensions and generators are very sweet.
So, given the dictionary {"a": 1, "b": 4, "c": 2}, if you want ["b=4", "c=2"], you can do this in Python:
d = {"a": 1, "b": 4, "c": 2}
l = [f"{k}={v}" for k, v in d.items() if v % 2 == 0] # replace [] with () if you want a generator instead of a list
var d = new Dictionary<string, int> { {"a", 1}, {"b", 4}, {"c", 2} };
var l = d
.Where(pair => pair.Value % 2 == 0)
.Select(pair => $"{pair.Key}={pair.Value}")
.ToList(); // remove this if you want an IEnumerable<string> instead of a List<string>
And if you want to sort the output by the dictionary value?
l = [f"{k}={v}" for k, v in sorted(d.items(), key=lambda kv: kv[1]) if v % 2 == 0]
# 5 3 2 1 2a 4 <- Execution order
var l = d // Execution order: 1
.OrderBy(pair => pair.Value) // 2
.Where(pair => pair.Value % 2 == 0) // 3
.Select(pair => $"{pair.Key}={pair.Value}") // 4
.ToList(); // 5What's wrong with its tooling?
IMO the biggest problem is that the Python libraries ecosystem is just too good to miss out on. I absolutely hate Python but even I have to admit it is much easier to do some stuff in Python compared to other languages simply because of the libraries available to do basically anything you could dream of.
Once you break compatibility stuff doesn't work… and a fast python that can't run my software is less useful than a slow python that runs the software.
I’m certain that there exists some subset of python for which this is true, but so what?
Several languages have persisted beyond their natural shelf life due to their library ecosystems. For a period of time, CPAN was one of the best arguments for using Perl. More recently, Java's ecosystem is also a pretty good reason to use Java right now, even if you don't like the language.
Languages seem to thrive early due to initial ease of use (Python, particularly, but also relative uptake of Go vs Rust), but can wane due to a thousand cuts (or sigils in Perl's case).
John Carmack seems to think otherwise.
> "That means that in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS"
https://twitter.com/id_aa_carmack/status/1503844580474687493...
The OP's comment is that while Python's native interpreter is slow, Python code as a whole can still be fast since people will delegate the hot spots to non-Python libraries optimized for speed.
To refute that, you linked to John Carmack quoting Horace He saying that Python's native interpreter does CPU-based math slower than a GPU. But...that's why people writing Python use libraries to delegate math-intensive work to a GPU, which is the OP's point.
The code that is written in a language other than Python is often not written by the developer writing the Python code.
As an example, I've been fiddling with inferring automatic data extraction rules for data in scraped webpages.
The meat of the algorithm and its heuristics is in Python. The heavy lifting of parsing HTML and evaluating CSS selectors is done by a C parser that I did not write. For me, prototyping the heuristics in C would have been quite a bit slower.
If you pick up Go expecting it to be a "more efficient Python", you will be sorely disappointed.
This is not the case in Go.
If you expand into just "kind of Python-like", you have languages like Nim that approximately fill the hole of "efficient Python".
My impression, though, is a lot of the popular frameworks/libraries for Python lean heavily on runtime reflection in Python (e.g. Django). So people end up stuck to the full Python interpreter/runtime.
In Ruby, parens are optional for all methods, and only required if you need to force non-default precedence.
Returning the value of the last expression seems reasonable to me (just like a UNIX shell).
In Python, I'm bothered by the little inconsistencies. Why is str.len (or length, or size, or count, etc) not a method? Why len(str) instead?
And Python has plenty of oddball syntax: significant whitespace, triple quotes?? You get used to it of course.
Ruby and Python are about equally efficient -- if you're optimizing for code efficiency, both are bad choices. In many domains, that's not a critical metric, so we talk about developer efficiency. I'll argue that developers are more efficient when they are more happy, and for me that selects Ruby by a mile over Python.
But I won't pretend that it's more than a matter of taste. :)
Yes. Especially this. The whitespace thing in and of itself has kept me from ever developing a fondness for Python. Clearly there are plenty of people who aren't bothered by it. De gustibus.
They are both dynamic, OO scripting languages, whose main implementation has a GIL, but beyond that, they aren’t all that similar.
> However, it’s kind of like Python’s weird brother,
Methods-with-self-in-args-list isn’t the “weird” one?
> with optional parentheses for calls without arguments
Parens are optional in Ruby on method calls and method definitions except where necessary to resolve ambiguity (including in the newer special single-line method definition syntax, its not a special syntax option for no-arg methods
> auto-returning the result of the last expression
Ruby is basically an expression-oriented language [0], everything returns a value, and a sequence of expressions returns the last expression in the sequence. Explicit “return” lets you short-circuit that. Python is a statement-oriented language, and statements don’t return things except an explicit return tells the function its in to return something. Both are common kinds of languages, just very different. Neither is “weird”.
I agree that it's uncommon, but I do like the clarity it adds compared to something like "this" which magically refers to the current instance. Hasn't C++ just added something like it with "deducing this"?
Compile times are not the 25-50 ms of a Python interpreter startup, but they are similar to the 250-500 ms of a Go or D compile, at least for small light on metaprogramming loops kinds of programs, and with, say, the TinyC tcc backend.
Of course, big frameworks take a long time to import (or to compile). So, your comparison mileage may vary. And once "ready to go" and finished a Nim program can start up in sub-millisecond time like any C program (esp. if statically linked).
Reimplement a large enough, commonly used subset of python stdlib using this dialect and we may be in the business of writing cross platform apps (perhaps start with android and Ubuntu/Gnome)
In the majority of use cases, your runtime is dominated by I/O, and for the remaining use-cases, you either have low-level functions written in other languages wrapped in Python (numpy, etc.) or genuinely have a case Python is a terrible fit for (eg. low-level graphics programming or embedded).
Why bother making a new variant language with limitation and no real benefit?
https://twitter.com/id_aa_carmack/status/1503844580474687493...
In any case, if your program is waiting on network or file I/O, who cares whether the CPU could have executed one FLOP's worth of bytecode or 9.75 million FLOPs worth of native instructions in the meantime?
The post is highly relevant. Next time, if you don't understand why, just ask.
More specifically, overhead (Python + pyTorch in this case) is often a bottleneck when tensors are quite small comparatively. It also claims that overhead largely doesn't scale with problem size, so the overhead would only matter when running very small tensor operations with pyTorch with a very tight latency requirement. This is... rare in practice, but it does happen to occur, then sure, that's a good reason to not use Python as-is!
You can open some software such as htop right now, and it will show how much CPU time each process on your system has actually used. On my system the vast majority of processes spend the majority of their time doing nothing.
Is it true for all software? Of course not! Something like my compositor for example spends a lot of time doing software compositing which is fairly expensive, and it shows quite clearly in the stats that this is true.
Another use case was parsing proprietary, text-based file formats with ancient encodings. I believe I did use Python for that as there wasn't that much data to convert anyway and it just worked.
Personally, I think going all the way to Nim [2] is more satisfying than a "gradually typed" system, though.
The biggest strength of Python is an ecosystem built on a blend of python code and native extensions, leveraging Python’s strength as a glue language. Dumping the runtime while remaining compatible with the extension ecosystem is quite a challenge, as is adding static typing and taking full advantage of it while also maintaining compatibility with existing ecosystem of code which is a mix of untyped and typed in the existing (optional) type system.
Which is probably why most “more efficient” Python efforts don’t try to do both of those things. Some did, or at least replacing the runtime, but most of them fizzled, not because they didn’t offer benefits, but because burning access to the existing ecosystem wasn’t worth those benefits. OTOH, what we see now is efforts to improve the base interpreter while maintaining compatibility, and opt-in compilation tools that can be used within Python, with a subset of Python or a statically-typed Python-like language that builds to extensions for the Python runtime. Examples include mypyc, Cython, Numba, etc.
Give me at least ksh and awk, they don't use much resources and are infinitely handy.
It's just one paper of course, but the energy results weren't too surprising. C was the baseline. C++, Rust, and Java close behind. Languages like C# and Go near the middle, and then Python, Ruby, and Perl at the bottom for most energy used.
[0] https://greenlab.di.uminho.pt/wp-content/uploads/2017/10/sle...
I do agree that there is a point where the scale tips. I wonder where that is, and what kind of application we’re talking about then. Many applications are at their basic level old corner shop crud, really.
Do you feel the same way about (e.g.) modern mechanized farming compared to classic farming that used human labor (generally unfree and unpaid labor, such as serfs or slaves)?
"Efficiency" is not the only goal that matters.
Comparing manual labor to industrialized farming is comparing (paper) accounting books to programming.
Whether you add gps based differential seeding to just doing it by ‘feel’ but still with machines is the difference of programmer brain cycles. Efficiency is not all obviously and there are different efficiencies one can find relevant. Why is this controversial?
Your argument is that programmers should use less-convenient and more painful languages to save energy.
The argument that agriculture should be performed by unpaid peasants or slaves is exactly the same.
Edit: it doesn't actually matter if they are paid or not (although they generally weren't).
The history of human progress is pretty much defined by using energy to replace human scutwork.
I'm not advocating for it, but it's an interesting thought experiment.
Might be nice for embedded devices though, but they already run C/C++...
Generally I’d expect less efficient languages to be more user friendly (well, a language which has neither advantage is just bad, why is anybody using it?). So, hypothetically these companies should get away with spending fewer programmer-hours on things. They won’t have to run the coffee pots, and air conditioners as late.
…A silly proclamation, of course.
99% of programming projects aren't "web scale" and the development costs far outweigh the operating hardware costs.
So while Python might be good for some stuff, please, please do think twice before choosing it for GUIs or other kinds of apps that should be snappy.
The GUI for pgAdmin4 isn’t in Python. Between 3 and 4 pgAdmin didn’t just change from C++ to (largely) Python with otherwise similar architecture, it changed from a classical desktop app to a web app with a Python backend (so you could run it on machine close, in network terms, to your db server, to which your workstations would have only an HTTP, not a pg protocol, connection, without maintaining a separate code base for that and desktop use.)
Never needed anything else.
That's it. Having cargo download 1000s of package from who knows who for basic functionality is unacceptable. Rust's features, while absolutely phenomenal to work with, just aren't worth having insane npm-style dependency graphs and all the risks they entail. In comparison, Python's built-in modules are a godsend.
So all in all, I think you should continue to stick with python.
I think python vs rust is always going to be a tradeoff either way. I just wish we had the best of both worlds in one language. Rust's base language with python's extensive library. Perhaps a new language will emerge that can satisfy both requirement now that Rust has a somewhat stable "safe" FFI. One can only wish.
Breaking up with Python - https://news.ycombinator.com/item?id=34177703 - Dec 2022 (172 comments)
Particularly annoying in Python is running external programs (yes, I know there are libraries) and manipulating structured data (list comprehension is not aligned with how I think, which is map() and filter() and lambdas not limited to single expression).
something
another
something}}}}} another
isnt much clearer anyway imo.
(Brackets also have the same problem.)
I'm in the IF
What am I in?
I'm in the if
}
}
}
What am I in?
In either case, the aligned visible lines above (presuming reasonable block size) will be what clarifies that, assuming correct indentation (which is necessarily the case with syntactic indentation) and the brackets just add (in the example) three additional lines of separation making that harder.
C# actually has great comprehensive defaults, they're just not enforced as strictly as they could be.
Building tools multiple people will be using regularly? Might be worth speeding up at least the slow parts with another language. (I don't know it well enough for it to by my first choice here).
Also: I appreciate typed languages more when working with a team. I'm not familiar with the options Python offers to keep teams on the same page.
python is not a good fit for anything I do these days
Yes, the drawbacks are here (async is not as good as in Go, typing I would say is rigid, packaging is not perfect at all). But I have a lot of projects where it just doesn't matter.
I have the feeling that its user base is pretty lopsided, but I might underestimate its importance in other areas, like teaching programming. I'd love to see a breakdown of its popularity by field.
Facebook + Instagram (largest deployment of Django)
Spotify (millions of lines of Python)
Netflix
Dropbox (millions of lines of Python)
YouTube
Plenty more examples.
How is this an answer to the question?
... do I really need to explain this?
P used for something in places in parallel of hundreds of (several exotic, rarely used) technologies is not really predicting how much P was used without scientific and ML uses. What if P is mostly used for scientific and ML purposes in those places, or as an experiment, study, not necessarily being a backbone? Or with other secondary agenda, overstated, etc. The mere fact that P is used in a huge amalgam of A,B,C,... tells very little alone concerning the question. Almost nothing.
Yes, needs some unwrapping what you believe.
I would argue moving away from Python at global scale doesn't even negate my point. All products change with scale.
Instagram (Cinder), Google (Unladen Swallow) and Dropbox (Pyston) have also all experimented with heavy engineering investment trying to improve Python performance and only one of them (Cinder) has outcompeted the “just rewrite it in a faster language” strategy.
That’s tail wagging the dog. ‘Why Python became the scientific computing language?’ is the right question.
Some guesses:
- Because scientists aren't that much into programming and Python is one of the easier languages to start with?
- Some aspects of its language design, like the syntax for slices, making it easy to express often used operations in a rather intuitive way?
- The two points above leading to the numpy project, enabling high-volume computations with a friendly programming language?
- Maybe Jupyter Notebooks played a role as well? It's a great teaching tool.
both can do async, both have libs for multithread workloads.
I wonder how critical the language itself is as far as multi-threaded goes.
I especially enjoy the types. Coming from Typescript and being quite a fan of the "dynamic language with optional typing" pattern it was quite cool to find that Python just has that.
Sadly quite a few pieces of software are likely going to stick with Python 3.7, so we are probably going to have to stick with the old syntax (explicit imports from typing and uppercase types) for a long time.
Now they have plans for a whole set of simple webapps. It's like a new lease of life!
It would not run on a $5 VPS from DO, because to deploy it required more memory (two gigabytes I think... mēh).
Mailman 1 ran in 40K. It was written in Perl. Clunky, not flash, and functional.
Mailman 3 is written in Python using Python tooling - Django I believe.
Clunky, flash, and functional. And costed my $5 a month extra.
I do not hate Python, I forgive the language for people misusing it. But it is a scripting language. That is what it is good for. Calling other systems. Not building the tight loops
I have Django projects, some of which I am going to assume are more complex than Mailman, using far less than two gigabytes in production. Not through any extreme optimisation efforts. It’s just…what’s happened. In my experience, the resource footprint you experienced with Mailman is not representative.
Notwithstanding the fact that we’ve seen amazing performance improvements in Python as recently ad 6(?) months ago, if speed were my primary concern, I am obviously not picking Python. This doesn’t mean that it should be relegated to the frankly at this point outdated bucket of “scripting languages”. The language’s features, performance profile, and package ecosystem all surpass these use cases. The conventional wisdom, which I completely agree with, is that these days things are good enough that speed need not be a primary concern for most people most of the time. Sure, oldschool Mailman was fast, but did anyone want to or know how to actually work on it anymore? Evidently not.
That sounds like frontend was written in something requiring a webpack or similar Node tooling, or you also ran a database on the same server.
In my experience, small Django setup with PostgreSQL can run fine on droplets with 512mb of ram, provided the traffic is not too heavy.
Add containers or anything node-related to the mix, and it'll be 2G+.
No Node.js or other nonsense.
Did use docker (another example of making simple things more complicated... )
However, we still use Python for APIs and Web Apps. No language is better than another, some are just better for one solution more than another.
That’s his quality but also the biggest problem. When you have something that cross the boundary python/c it get painful as you have two language to jungle with the handling of packaging, compilation.
Otherwise I fear it will be Javascript that will ultimately conquer everything.
Python apologists seem to be looking for excuses to avoid the hard work of learning proper programming skills.
Obviously, some languages are better than others for specific use-cases and one should think through the strengths and weaknesses of each before beginning a new project (assuming you expect the project to have some longevity). Similarly, it definitely is possible to make the wrong choice about which language to use. But even as a PL "enthusiast", I've learned over ~15 years of leading successful projects and teams, that evaluating languages is always team/org/business-dependent and the considerations that factor into those contexts are distinctly different from the kind of considerations that I, as a PL enthusiast (and I suspect many others in the HN crowd) would prioritize. You have to consider things like:
1. What stage in the business in, i.e. early stage where you may simply want to get something, anything working and it's viable to do so in whatever language will get you there fastest (warts and all, tech debt be damned, etc) vs the critical juncture where you're beyond finding product-market fit and you now have a business that is working and you need to increasingly focus on reliability, scalability, etc?
2. What does the pool of engineering labor in the market you have access to look like? For example, it may be the case that you'll have difficulty, for one reason or another, finding developers who work with language X, but could easily find many who can work with Y (of course, good devs should theoretically be able to ramp on any language, but whether you'd want them to do depends on how crucial using that particular language is to your venture, whether you can afford the ramp-up time, etc.
3. Also somewhat related to the quality of engineers you have access to, the community/packages/ecosystem surrounding the language may matter a lot to such a degree that even if some other language were a better fit from a purely technical perspective, it may leave you to have to fend for yourself in ways that your team simply isn't up for.
.. and many more considerations.
Beyond all of this, I resonate with the author's orientation of wanting the language to just get out of the way. Sometimes you're looking to geek out on a stimulating and interesting language. Other times, that's the last thing you're thinking about. Bonus points if the language you find most compelling is also the one that, for reasons like the ones mentioned above, is the one that makes the most sense for you and your team. All in all, the point is that assessing a language in isolation, without any regard for what you're actually trying to accomplish, what your team looks like, etc. doesn't make much sense.
Side note: I do wish more Substack authors would allow comments even from non-paid subscribers. I could understand restricting it if you already have an established paid subscriber base and a chatty comments section. But if comments are non-existent, it seems to me like the author is potentially losing out on valuable feedback/commentary by restricting commenting to paid subscribers.
I think it's worth learning a lisp, a functional language, a logic pl, or something from some other paradigm you don't typically use. Maybe do a side project in it. The lessons you learn will surely make you a better programmer overall even if you don't maintain your mastery of it
The “Bruce Lee” of programming would definitely use assembler and deeply understand computer architecture at a fundamental level. The idiom of Bruce Lee is someone who is hyper passionate about their craft.
I consult as an X "expert", but have worked, and do work, extensively with tools outside of X's scope.
You might think that C would be a contender for a long-lived language, but what about a world where humans don't do the code writing anymore?
Languages are scaffolds. You need them for a time, then you build around it and are onto something new.
Everything changes. Time will ultimately wash all that we consider permanent away.
Both are, for all practical purposes, immortal languages at this point. You correctly point out that they aren't guaranteed to survive the next technological mass extinction event, but I think that might just be a truism.
The same holds for music, art, photos.
Assembler, C and shell script have been and will be around for many years.
As an end user, i.e., someone who runs programs more than she reads/writes/edits them, I find Python's startup time is just too slow. I have a folder full of tiny, fast shell scripts and C programs that I rely on year after year. Replacing this with a folder full of Python scripts is difficult to envision.
Assembler, C and shell script will still be ubiquitous and foundational no matter what subsequent programming language hype comes over the internet. Boring stuff that works. I am typing this comment from a program written in C.
"The industry" (cf. the end user) seems to have endless time to burn because the stuff it promotes through hype always presents a massive timekill to those who embrace it. A conspiracy theorist might imagine that the industry is trying to keep itself busy and maintain peoples' attention. Regardless of intent, that seems to be the end result.
It is like that hardware company that wants people to keep buying new computers every few years, and tries to prevent/discourage anyone from repairing and retaining them.
I don't think the Blub essay is about learning a constant stream of new languages. I mean, the author of the essay doesn't seem to be constantly learning new languages.
The Blub essay, whatever its flaws may be, is about people being unaware of languages and features more advanced than their tool of choice, and finding them strange. Or in other words, that it's easy to see how poorer tools are more limited than the ones you're using, but very hard to understand how better tools are better, or even acknowledge they are better at all.
If your business is trying to solve hard problem X and your best idea is to use this tricky advanced language feature and then apply these compiler flags… then you’re going to lose to the person who knows of a programming paradigm (e.g. logic programming) unlocking a simpler solution or a data structure unlocking a more performant solution.
That sounds a bit abstract. To make it more concrete - imagine a problem where you need to apply a function in concurrent windows over a time series but you’ve never experienced any of the array programming languages.
(Edited ;)
It's the manipulation of data that makes me enjoy programming.
Python ? Not fun at all. Try list comprehension. You read it from right to left ?
Or process data, not by left to right (pipeline), but in a messy order.
One liner lambda ?
Indenttion, self, pip hell.
Too many small quirks which force me not to use Python for "fun".
For serious stuffs ? Maybe python is the best choice now.