Python Is Eating the World
techrepublic.com
techrepublic.com
I default to python for scripting at work because it is popular - we are more likely to find folks who have at least seen it before. Most of my throwaway and experimental code is Perl5, just out of long habit.
It "downgraded" regular expressions, which are entrenched as first-order citizens in most of unix, and made you use function calls instead.
Although it would seem to make regular expressions clumsier to use, it made it easier to encode and decode them.
One of the most frustrating and painful things is trying to figure out if something in a regular expression is a literal character, a matching character or an escape character, and it can be deeper, such as a variable or other syntax character from the enclosing language. This kind of stuff trips up beginners and seasoned folks alike.
Python did the same thing with path manipulation, which suffered from the same sorts of problems.
Also the ellipsis literal `...`, never need to use `pass`.
[0] https://docs.python.org/3/library/typing.html [1] http://mypy-lang.org/
It's also slow, but I can deal with slow. Machine time is often cheap, while developer time is expensive.
> It's also slow
It looks to me like you're describing Nim. Static typing, similar syntax to Python, expressive, readable, powerful, fast; and destructors are being worked on as we speak.
Having worked on a python codebase for the past eight years, static typing isn't really much of an issue. Just document what types you expect, and write the code to match. If it runs, it runs. Plus, you have pylint to catch most of the silliest mistakes.
Python has finalizers rather than destructors, though they are sometimes referred to as destructors (the Python docs explicitly note that this is incorrect but not why, there is discussion of the difference—inbthe context of C#—here: https://blogs.msdn.microsoft.com/ericlippert/2010/01/21/what... )
$ python -c "import this" | grep purity Although practicality beats purity.
You probably meant to write "statically typed"
Python doesn't have just hinting, it has an optional static typechecker.
Anyhow, I don't see it's weirder than static languages adding the option to use “dynamic” as a pseudotype to opt out of static typing on a case by case basis.
Languages are more than typing regimes, and language designers can believe that there is no one typing regime that is right for all use cases.
I watched as the ML space embraced it as the lingua franca. It was exciting.
Best decision of my career :)
I feel the same way as you do about python about C#/.NET, and I'm happy to have picked C#. I think for most things it would be as good as Python, e.g. web-dev, app-dev. With Xamarin it's a great choice. For data science I'd have to go running to Python or R (maybe I'd do as much in C# as possible), but luckily I don't do much data science.
i would agree that there is an amazing amount of productivity to be gained by reaching for mastery over a language.
that said, the efficacy of that mastery is determined by how "useful" the language is in a wider context. this is, for better or worse, highly influenced by "popularity" when it comes to programming languages.
Mastering something like brainfuck? ymmv ;)
I can think of a few specific projects in my early career where I wrote things in Python while other teams in my company were trying to write similar things in C++.
The Python implementations (which used Cython for a few tiny performance critical sections) ended up running much, much faster than the C++ projects (which were written by very veteran C++ engineers at the time). More than that, they were easier to test, easier to modify, and easier to teach to random new programmers joining the team. We benefited from much more user-friendly third party tools and specialized domain tools and could easily wrap any of the in-house & domain tools only accessible in C/C++.
I’m honestly really surprised. Can you provide some more detail as to how Python was beneficial for these?
Cython is a slight superset of Python that compiles to heavily optimized C or C++. So it is extremely easy to write & autogenerate Python bindings to your own C extension modules.
The difference is that it frees you to write Python (much easier to write & maintain) for everything, and only care about compiling down for speed in a few isolated places. Contrasted with writing everything in C/C++, this is a big advantage.
I've been writing Python for over a decade and I find this a bit surprising (not saying you're wrong). Python's strong suit was always speed of productivity over runtime speed IMO.
Python (CPython anyway) is nothing but a special DSL for writing C programs. Any part of Python is absolutely as fast as C if you want it to be.
In the projects I’m referring to, we just used Cython for this. Instead of wasting time writing in C for the whole project, managing memory allocation, building up abstractions over structs with macros, or using these things from inscrutable C++, we just wrote simple Python, used kernprof to measure places where running time actually mattered, and targeted those few small places with a Cython extension module.
Since the small part of performance critical code was much simpler in Python (using pretty much zero abstraction / functions only in Cython), the compiled Cython code was faster than corresponding components of C++ implementations (that took much longer to write and understand).
Most of the rest of e.g. Python standard library calls in the surrounding glue code is already C-level extension functions in Python with no measurable running time difference than C itself, and the rest of the custom Python code was measured in the profiler to be fast enough so as not to matter.
The “slow” parts of Python come from the object data model and a variety of protocols like iteration, properties and MRO. But it isn’t really fair to call it “slow” because it’s a trade-off to get those features. If you need an object that supports being iterated, having its string representation overrided, and inheriting methods from some mix-in, then you are agreeing to pay a running time cost for this. If you don’t need that stuff, you could just write what you need in e.g. Cython. With more heavily automated tools now like numba, this argument for using Python as a first choice for writing very fast code is only getting more mainstream.
Most of the time, you do need a bunch of “batteries included” features of Python and running time is not the resource constraint you’re worried about. And in the occasional exceptional cases when you need raw speed, you can use extension module strategies (e.g. how numpy was made) to be as fast as C.
For instance lists are dynamically allocated, so repeatedly appending to the list may cause it to reallocate.
lst = []
for i in range(10_000_000):
lst.append(i)
However due to the leaky abstraction, you can force allocation all at once which speeds things up, albeit at the loss of clarity.
length = 10_000_000
lst = [None] * length
for i in range(length):
lst[i] = i
However in these trivial cases Python offers more idiomatic approaches, which also perform better.
lst = [i for i in range(10_000_000)]
lst = list(range(10_000_000))
Strings behave similarly when comparing the join method to incremental appending.
string = ''
for _ in range(10_000_000):
string += 'a'
string = ''.join('a' for _ in range(10_000_000))
string = 'a' * 10_000_000
Meanwhile, interesting stuff is being done on Python, Julia and R and Rust , if you also include system programming.
Python didn't even reach the fame Perl had in the 80s to mid 2000s.
Python is for beginner programmers, who absolutely don't want to learn anything new at all.
Python is used by the largest companies as well as the smallest one person company who need to get shit done. You can continue liking Java, without becoming bitter about Python.
It was not the best decision of my career :-/
I was already firmly invested in Java, another good choice.
Another common choice was Ruby, probably not so good.
Oddly enough it started at home. I was a long time Visual Basic programmer, but I decided to try Linux on my home computer, and needed a way to write simple programs. Going a step further, I decided that I wanted my tools to be open source and platform independent. After evaluating a few tools, I settled on Python and started playing with it. I tried out things like Maxima and Octave as well.
A few months later, a big crisis landed in my lap at work, and I decided to use Python to develop the solution. I was amazed by how productive it seemed to be, and I decided to sweep more projects towards the event horizon of Python.
I've also climbed aboard the "reproducible science" movement. Though I don't publish my work, reproducibility is just as much of a problem within a closed setting. Jupyter notebooks have made it much easier for me to dredge up an old project and figure out what I did, months or even years later.
The overwhelming majority of code I work with is in Python and I can't remember the last time I encountered an issue caused by invalid types. Static analysis tools, good documentation, and peer reviews (things that should be part of any critical code development) almost always prevent those sorts of issues.
Strong typing is great. Just don't exaggerate its effectiveness.
A type system is a good way to get static analysis and documentation :)
Which is the main advantage of python. Since there are heavily optimised libraries for everything from optimisation to numerical analysis to ML, 99% of people don't need to attempt to be a Real Programmer and write the hand-optimised Fortran/C/assembly code themselves.
I understand the benefits of static typing. I'm just tired of it often being so overstated.
Use a Ferrari on a racetrack, use a fortified tank on the battlefield.
The same with something like JIT vs compiling, I don't want to find out that some edge case a quadrillion loops in has some simple error a compiler could have found.
I’m a much faster developer in languages that are able to produce decent documentation about their typed interfaces, than I am in languages that are much more loosely defined.
There’s an amount of uncertainty in interpreted languages that always leaves me questioning the code—uncertain that it handles the cases I think it does.
If you had to write this as a feature in a maintainable way, that's when python would start to suck
All languages are typed in one way or another. Unless you write a generic function, you cannot evaluate (+ 2 "a") in scheme. Here is racket's error message:
> (+ 2 "a")
; +: contract violation
; expected: number?
; given: "a"
; argument position: 2nd
; [,bt for context]
>
If the code wasn't so obvious as the above example, you'd have a hard time mentally working out the correctness. E.g., given both (shoot camera person) and (shoot gun dummy), you'd not want to accidentally pass in the wrong arguments like (shoot gun person).
https://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic...
I cant really pin it down, but when I am programming in Python, the language flows along with my thoughts.
During intensive programming sessions, I never really feel the difference between my thoughts and the language flow.
I hope I make sense.
Often, I start solving a problem, solve a core part, then have to tack on additional conditions or workflows and I can just think about them and build them in python.
I have never had that happen in other languages and I have used C#, JS, C, C++, Java, Scala & VB.
Talk to a proficient C++ programmer and s/he'd say it flows.
It's easy it get tunnel-vision when one is only thinking of one's own experience. It might differ for others. I didn't mean to undermine your comment though. Just a thought.
But the thing is that the flowing part came effortlessly with Python. Maybe I need a few years with C/C++/java etc, but with Python, it took me less than a couple of months.
My thoughts could be very easily expressed as working code.
However, I noticed the same thing happened for other people, BUT they didn't think like I did.
Some used different ways of expressing the same thing, such as "if not" vs "unless". (although I do like unless)
When I learned python I thought it was clumsier. It was harder to use regular expressions, and didn't have shortcuts like:
while(<>) { $a = 1 if /abc/; }
Also, trading (it now seems) unnecessary syntax items like braces and semicolons for syntactic indenting has made the language much more compact on the screen and therefore more is readable at a glance.
Most definitely. I hate typing those colons and brackets.
Python has a much simpler grammar than most programming languages.
Also, I use spaces, never tabs!
As for screen real estate, I couldn't care less. What I do care more about is not wasting time making sure indentation levels are correct after a nasty merge, or fat fingering something that changes the logic of a statement, or not being able to easily move a block to another or not being able to visualize which blocks are collapsed or not due to a lack of separation, etc.
I've spent plenty of time in languages that are whitespace sensitive (python, haskell primarily) earlier in my career, but I would never go back.
Is like saying I know Python because I've worked with Django. Yes, I'd know the Python syntax and I'd be able to read Python code, but in my opinion I'll really not get to know the soul of the language
Rails is definitely Ruby, though not all of it. Non-trivial Rails work requires non-trivial understanding of Ruby, though there is some of the language and more of the standard library you can probably get by without in most Rails work. That's true of work in any application domain, though.
OTOH, Rails is highly opinionated in ways which aren't universal in Ruby, so it's quite possible for Rails to not click but Ruby to be a good fit for a particular programmer. I personally like Ruby as a language more than I like Rails as a web framework. Of languages of their general type, I go back and forth between preferring Ruby and Python.
My comment on Ruby language comes from the learning Ruby part.
If there's a new hotness in python web frameworks (or whatever) that suddenly everyone wants to use, questions about that will top the list of new questions.
Python has immortalised itself in ML because it's extremely important to be easy and clear if you want to write already complicated data science, and it has the data structures that other languages just can't compete with.
Plus it's very close to pseudocode, and a lot of ML is spoken in algorithms.
mind elaborating?
It is seriously one of the best places in the world to grab some popcorn, hangout, and watch beginners, students, CEO’s, experts, consultants, etc rant and rave about company X or programming language Y.
Genuinely love the spirit of HN, even if people say it’s different for the worse now.
I assume there aren’t many places on the internet for nerds to publicly hang out with experts and executives in their field.
On another note, many of these arguments talk about typing. Why say a language must be static or dynamic, etc, when you can achieve both. Making a C# MVP is much more verbose than python. Maintaining an untyped 3 million LOC monolith, is likely harder than C#.
Everyone here is working on pretty different projects and codebases, so why argue for one?
Anyone have gripes about mypy, or experience successfully using it(or an alternative)?
I would see really interesting questions shut down with:
"this question will likely solicit opinion, debate, arguments, polling, or extended discussion."
I wondered why they couldn't be inclusive and possibly just move the questions elsewhere but let them live.
(This is coming from someone who loathes Python, btw.)
Python isn't just "well, it's the only thing there" like C++ was.
Python already had a well-established and well-liked competitor that it had to displace--Perl. It is extremely hard to communicate the juggernaut that was Perl/CPAN back about 1996. So, Python has already demonstrated that it has better survival traits than most of its predecessors (Tcl, Perl, etc.).
The current popularity of so many of the languages developed in the 1990's shows that they really did do something significantly better than the languages that came before them.
"It's the developers, stupid"
And also, perhaps, recommend something for pure beginners to the language? I know C/C++/C#/Obj-C, Java, Swift, JS, PHP, et al, to give some context.
For example, if I have to design software for a micro-controller that uses C/C++ I sometimes write down the algorithm in Python (with editor support: auto-formatting, etc.), then I implement it in C/C++. There is no better way to write pseudo-code that I know of.
Python is also great to create a prove of concept for something. It allows for rapid prototyping due to the simple syntax, the high-level features, the huge ecosystem and the dynamic type system. Most of the time the result is good enough for production. You can add type annotations and get statically typing via mypy [0] to harden the code base. This works really well. (See: JS->TypeScript).
Python has a well developed FFI which allows you to call C/C++/Rust, without large performance hits. I think this is one of the main reasons for Python's success. It can be used as a high level interface to a huge low level ecosystem. ML wouldn't exist in Python without that.
Python is a great language to create simple scripts as well. I use it as a bash replacement.
But beware: Python can get really ugly if you rely too heavily on OOP/Inheritance. There is a lot of ugly Python code out there. I think Python shines the most if you use it in a procedural/semi-functional style and describe data-types for your business logic (almost) entirely with (dump) dataclasses [1] and namedtuples [2].
Beware2: Python's packaging story (package manager, etc) is currently in a messy state. There are good solutions, but it's difficult to find the right tools for a newcomer. I think it will take a couple of years for the dust to clear, but you might want to check out pipenv [7] or poetry [8]
These packages will make your life a lot easier. I use them for every project:
- black [3] or yapf [4] (auto-formatter)
- mypy [0] (statically typing with editor support for VSCode)
- pylint [5] or flake8 [6] (linter)
[0] https://github.com/python/mypy
[1] https://docs.python.org/3/library/dataclasses.html#module-da...
[2] https://docs.python.org/3/library/typing.html#typing.NamedTu...
[3] https://github.com/psf/black
[4] https://github.com/google/yapf
[5] https://github.com/PyCQA/pylint/
[6] https://github.com/PyCQA/flake8
In comparison with Ruby, even after years I can write complex expressions and get standard library names just by guessing and be right. In Python it’s like my brain is wired at the opposite of every convention. Even JavaScript feels more natural.
I have 0 joy programming in Python even if I get the job done, whereas in Ruby (or other languages) I’m instantly in the zone.
I know it’s purely a taste issue but I’m sad Python “won”.
c/c++ are out there, crushing it. so is python and javascript and java and go and c# and so on and so forth.