Can you be more specific about how it increases your burnout? Is it the language, or someone forcing you to use that linter and settings?
It's definitely both.
Preface in true internet style: these are just opinions and you may not share them. That's fine.
I really don't like Python as a language. I don't like its total lack of composability. I don't like its over-reliance on a very dated vision of OO. I don't like how its list comprehensions aren't generic. I don't like how it farms all its data structures out to C. I don't like how it uses cruddy one line lambdas and forces me to name things that are already named by the things I'm passing it to.
And also the linter just exacerbates these things, because the linter is just a crystallized slice of a kind of software engineering culture I really don't like.
Python iteration is generic.
https://docs.python.org/3/reference/datamodel.html#object.__...
Please don't spread FUD.
> That's an implementation detail of CPython. I don't see how it's important for a language user either. Jython uses Java objects. PyPy uses RPython.
This strengthens my point though?
> Please don't spread FUD.
This is that culture thing I'm talking about.
> This is that culture thing I'm talking about.
You made a false claim. You referred to Python iteration as not generic, when what you really meant is that Python lacks first class monad support.
If you want a monadic programming model, you're not going to be a happy camper in the Python world. But I doubt that comes as a surprise.
If Python is so great how come it can't even express a decent data structure that it needs? I'll happily level the same criticism at Ruby if you like.
> You made a false claim.
Firstly: This is not High School Debate Club. There isn't some kind of point system. Language and expectations like this are not only counterproductive (in that they essentially turn every conversation into a series of verbal ripostes in which the goal is to be most right rather than *learn the most) they're also tedious and unwelcome.
> You referred to Python iteration as not generic,
I did not. I said list comprehensions weren't generic, and then I tried to explain my complaint. It may be that someone has done a legendary feat of going through and creating a bunch of mixins for some of the gaps in the list comprehension coverage such that you can find some way to override how non-determinism is expressed. If so, please point me to it.
I'm sure you know this is a deliberate design choice/tradeoff. It's arguably turned out to be a bit of a millstone for languages that eventually want to grow up to be general-purpose-ish, but that wasn't as obvious at the time.
I remember having similar conversations with the creator of Io and seeing the Pike mailing list have similar concerns. Hindsight is 20/20.
Why is this a requirement of a "great" language? By not requiring a language to be self-hosting, you are adding more degrees of freedom to your language's design, so I could even see an argument that writing it in C is an improvement. I don't necessarily agree with that, but I don't see why cpython written in C implies it is a bad language. Maybe you could elucidate your thinking?
Why do you think that? Languages/runtimes with high C integration and fairly exposed C bowels like Python and Ruby have, over time, turned out to be very hard to evolve compatibly.
Because it's a fact? With cpython you can develop things in python if that suits you or C if that suits you. You have more freedom to choose what fits your use case. I'm not saying this is necessarily good, but I don't think it's obviously bad. I'd like to hear from people who think it's the case.
> Languages/runtimes with high C integration and fairly exposed C bowels like Python and Ruby have, over time, turned out to be very hard to evolve compatibly.
That is true, but it's also arguably one of the reasons cpython became so popular in the first place. The ability to write C-extensions when appropriate has been very powerful. It's certainly caused issues, but I think if python didn't have exposed bowels it may never have become nearly as popular. What if numpy wasn't ever written? (This isn't to say that they couldn't have exposed a better C-api with fewer issues, but hindsight is 20/20...)
Who gets to be the language police? I'm fine with "High School Debate" but "verbal ripostes in which the goal is to be most right rather than learn the most" is not an accurate description of that.
But I won't engage with someone who does this the same way, because they're not engaging me as a human. Unless, of course, they're already dehumanizing me (as occasionally happens on this website) and then I don't feel quite so bad about it.
Are you really that lacking in self-awareness that you responded to someone annoyed about programming culture by going full "debate with logic, facts and reason" mode?
Like I said, it’s not the perfect language, it does not have Erlang’s pattern matching nor Lisp’s insistence on thinking about almost everything in a recursive manner, but you can at least mentally try to incorporate those two philosophies into Python once you know about them (and once you know about other similar such great programming concepts).
List comprehension:
a = [x for x in stuff if x > 1]
a = list(x for x in stuff if x > 1)
a = MyCustomCollection(x for x in stuff if x > 1)
If you're having a burnout because the PR you just opened are being rejected by the linter you probably just make the linter apply the modifications automatically in pipeline.
We used to hate the linter checks in my current workplace because it was really boring to fix the issues. Now the CI simple fix them, and nobody cares anymore.
I believe Python is quite possibly the best language for a few things
* Exploratory programming - such as what data scientists do
* Writing programs that will never grow beyond 150LOC, or roughly what fits on a screen + one page down
When I have those two constraints met I am almost always choosing Python.
Here are some problems I face on codebases as they scale up:
* Python conventions lead to what I consider bad code. Devs will often choose things like 'patch' over dependency injection, and I have seen on multiple occasions projects derided for providing DI based interfaces - "oh, why would you write code that looks like Java? This is Python".
There's a lot of death by a thousand cuts. Keyword args in functions are abused a lot, because they're "easy", ability to patch at runtime means it's often "easy" to just write what looks like a simple interface, but it's then harder to test. Inheritance is "easy" and often leads to complex code where behaviors are very, very non-local.
Dynamic types mean that people are often a little too clever with their data. I've seen APIs that return effectively `Union[NoneType, Item, List[Item]]` for absolutely no semantic reason, without type annotations, meaning that if you assumed a list or none came back you were missing the single Item case. The implementation actually internally always got a list back from the underlying query but decided to special case the single item case... why? I see this sort of thing a bit, and other languages punish you for it (by forcing you to express the more complex type).
* I find myself more and more leveraging threads these days. I did this often with C++, and all the time in my Rust code. Python punishes you for threading. The GIL makes things feel atomic when they aren't, reduces you to concurrency, all while paying the cost of an OS thread. Threading primitives are also quite weak, imo, and multiprocessing is a dead end.
And, really, Python is anti-optimization, which is a good and bad thing, but it's a bit extreme.
* Implicit exceptions everywhere. I see 'raise Exception' defensively placed in every Python codebase because you never know when a piece of code will fail. I see a lot of reliance on 'retry decorators' that just catch everything because you never know if an error is transient or persistent.
The common "don't use exceptions for control flow" is broken right off the bat with StopIteration. I just think error handling in Python is god awful, really.
* Mypy is awesome, but feels like a bit of a hack. The type system is great, and in theory it would solve many problems, but coverage is quite awful in open source projects and the type errors I get are useless. I actually only use mypy to make my IDE behave, I don't run it myself, because my types are actually technically unsound and the errors are too painful to work with (and honestly the type system is quite complex and hard to work with, not to mention the bugs).
There are lots of other smaller issues, such as my distaste for pip, py2/3 breakage, lack of RAII, the insane reliance on C all over the place, I think syntax is bad in a lot of places (lambdas, whitespace in general), etc but these are probably my main sticking points with using the language for large projects.
Again, Python is actually my favorite language for specific tasks, but I really think it's best optimized for small codebases.
That's not an idiom python has ever subscribed to though, it's always subscribed to the "Better to ask forgiveness than permission" - I think there are strengths to both viewpoints, but honestly I think "don't use exceptions for control flow" is more of a convention that a "truth"
In absolute terms or when coding on paper, perhaps. But in the real world and if performance even remotely matters, it’s as close to a universal rule all languages end up embracing or turning into creaking hulks of slow code given how exceptions work in practice at the level of cpu execution units.
C#/IL/.NET embraced excerpts heavily at around the same time (start of the 2000s) but in time developers (in and out of Microsoft) learned the hard way that it doesn’t scale. With .NET core, exceptions for flow control are completely verboten and APIs have been introduced to provide alternatives where missing. Exceptions should be so rare if you chart all handled exceptions, you shouldn’t see any and would thoroughly explore why one or more pop up when a system or dependency hasn’t exploded.
Having ported some decently large codebases from OCaml to F#, the heavy use of exceptions in OCaml (where exceptions are very lightweight by design) had to be changed to normal control flow with monads to achieve good performance in F# — specifically because of .NET’s slow exception handling.
Exceptions have two core properties: (1) non-local jump (carrying a value) and (2) dynamic binding of place to jump to. Contrast this with e.g. break / continue in loops where (2) does not hold. If most use cases of performant OCaml exceptions were not making use of (2) that would be an interesting insight for programming language design.
Have you got data on this matter?
let modify_opt a f l =
let rec aux p = function
| [] ->
(match f None with
| None -> raise Exit
| Some v -> rev ((a,v)::p))
| (a',b)::t when a' = a ->
(match f (Some b) with
| None -> rev_append p t
| Some b' -> rev_append ((a,b')::p) t)
| p'::t ->
aux (p'::p) t
in
try aux [] l with Exit -> l
That's, as you say, a statically scoped fast exit. One does not need the full power of exceptions for this (exceptions'd dynamic binding comes with a cost). If exceptions are widely used for this purpose, one might consider adding a nicely structured goto to the language. Something like linear delimited continuations?
For instance, the basic stream/iterator in Ocaml is Enum, which always uses an exception to signal when the stream has been exhausted, rather than providing a "has_next" predicate.
The catch site of this exception is dynamic.
Python explicitly does not agree with this...
The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.
OK, I'm willing to contend that that is the case. It doesn't have much to do with my overall issue with error handling.
> The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.
This is a very loose criticism of my post, I don't know how to respond to this. I've written Python for years, I think I gave it due credit for what it's good at.
I am not trying to write some other language with Python, I just think Python is not a very good language compared to others given a lot of fairly typical constraints.
I've written Python for years, I think I gave it due credit for what it's good at.
I'm not sure how fair this assessment is, considering the multitude of Python projects of considerable size and scope.
It would be very silly for me to say that you can't build large systems in Python, I've worked on plenty that are much larger myself.
Saying a language is possibly the best for two important use cases (exploration, small programs) is quite a statement, in my opinion. I don't think I believe that there's a language that excels so well in, say, building web services.
Whats the convincing argument for defending this? Ive always heard its just the way python is. Through experience using exceptions for anything other then exceptional situations and errors seems messy.
Imagine you are working through a series of EDI files and trying to post them to a badly documented Rest(ish) API of some enterprise system. If the file is bad (for whatever reason) you need to log the exception and put the file in a bad directory.
Pythons use of exceptions for control flow is perfect for this. If file doesn't load for whatever undocumented reason, revert back to log and handle the fallout.
"Oh I see a pattern, this API doesn't like address lines over 40 characters, I will add a custom exception to make the logging clearer, and go and try and see if I can fix this upstream. If not I will have to write some validation"
It is this dirty world of taking some other systems dirty data and posting it to some other systems dirty API that I find Python rules.
I have never worked on a large application where I owned the data end-to-end. Maybe there are better choices than Python for that?
I think StopIteration is being removed. That was a a bad idea.
In fact, the Python developers recently doubled-down on the design by introducing `StopAsyncIteration`.
PEP 479 is specifically related to generators:
> If a StopIteration is about to bubble out of a generator frame, it is replaced with RuntimeError, which causes the next() call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception.
I see the point that python invites these anti patterns.
But on the other hand, a software developer that returns Union(list[item], item]) in Python is probably also going to mess up a Java program sooner or later.
That said, the amount of cpu cycles wasted even with best of breed pandas is insane, and when you want to do something “pythonic” on big data it all falls down. When you want to deploy that model you are also going to have problems.
That said, it’s still the best tool for the job, but it’s certainly not because of the creators of Python.
Python without type annotations can be painful in an IDE - it chokes a lot on those features.
Since I don't take a very type driven approach to Python (it would be too slow since I'd have to figure out 3rd party library types and shit like that) I just write annotations in places where I personally know the type. Mypy complains about this for various reasons - probably because my annotations are not technically correct, because I'm not a pro at generics in mypy and working out inheritance, as well as general mypy issues like poor support for class methods and that sort of thing.
But I ignore all of those errors because the IDE can still work things out.
Why is multiprocessing a dead end?
The overhead obviously is still there, but the interface is a drop in replacement for threadpoolexecutor, which looks basically just like a multithreaded/async/future-based collect.
To communicate across mp you use pickle, which is a pretty significant performance impact relative to something like threading. There's also the issue of copy on write memory + reference counting interacting poorly.
I suspect this is the root cause of the difference in our experiences. My uses of mp have usually been somewhat more "embarrassingly parallel", for instance having a list of data elements which need to be processed with the same algorithm. For this use case, the usage of mp is pretty simple, often only a `pool.map(f, xs)`.
I can imagine that pickle might have tricky edge cases and/or be slow.
a) All reference counts after the fork are going to cause a copy of memory, so any memory access (even a read) can trigger a copy.
b) Even to send 'xs' over you must serialize it via pickle, and then the callee must deserialize it
That may be fine for your use case but it's strictly worse than just sharing a reference across a thread.
1) No pattern matching. In 2019, this is just unacceptable. Pattern matching is so fundamental to good FP style that without it, it becomes almost impossible to write good and clean code.
2) Statement vs expression distinction. There’s no need for this and it just crippled the language. Why can’t I use a conditional inside an assignment? Why can’t I nest conditions inside other functions? It makes no sense and is stupid and inelegant.
3) Related to 2), why do I need to use a return statement? Just return the last expression like many other (better) languages
4) Bad and limited data structures. In Python all you get are arrays, hashmaps, sets. And only sets have an immutable version. This is unacceptable. Python claims to be “batteries included” but if you look at the Racket standard library it has like 20x more stuff and it’s all 100x better designed. In Scala you get in the standard library support for Lists, Stacks, Queues, TreeMaps, TreeSets, LinearMaps, LinearSet, Vectors, ListBuffers, etc.
5) Embarrassing performance. Python is so slow it’s shameful. I wrote a compiler and some interpreters in college and I honestly think I could create a similar language 10x faster than Python. Sometimes you need to trade off performance and power, but that’s not even the case with Python: it’s an order of magnitude slower than other interpreted languages (like Racket).
6) Missing or inadequate FP constructs. Why are lambdas different from a normal function? Why are they so crippled? Why do they have a different conditional syntax? The only sort of FP stuff Python has is reduce/filter/map. What about flatMap, scanr, scanl, foldl, foldr? Or why doesn’t Python have flatten? All of these are very useful and common operations and Python just makes everyone write the same code over and over again.
7) No monads. Monads can be used for exceptions, futures, lists, and more. Having to manually write some try catch thing is unseemly and worse than the monadic Haskell or Scala approach.
8) No threads and no real single process concurrency. Despite Python being used a lot, no one really seems to care about it. How can such a problem not be solved after over 20 years? It’s shameful and makes me wonder about the skill of Guido. There’s no reason why Python couldn’t have turned into something beautiful like Racket, but instead it has been held back by this grumpy old guy who is stuck in the past.
9) Others might not have a problem with this, but I detest Python’s anything can happen dynamic typing. It makes reasoning about code difficult and it makes editor introspections almost impossible. If I can’t know the exact type of every variable and the methods attached to it, it hampers my thinking a lot. I use Python for data science and if I could just have a language that was compiled and had static typing I would be 3x as productive.
Let me conclude by saying there currently is one good reason to use Python: if the domain is ML/DL/quant/data science Python is still the undisputed king. The libraries for Python are world class: scipy, sklearn, pandas, cvxpy, pytorch, Kerala, etc.
But Julia is catching up very fast and the people I have talked to are getting ready to ditch Python in 2-3 years for Julia. I don’t think I’ve encountered anyone who didn’t prefer Julia to Python.