Hacker News Reader

Comment by stevesimmons | Hacker News Reader

Python's been my main language for 22 years now.

Whenever I think I should switch to faster, more efficient language (C#/Go/Rust/...), I am repeatedly blown away by what Python lets me do, with an incredibly ecosystem of packages, and all the ergonomics of incremental development from a REPL.

Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement.

That's broadly comparable to the time for load the source file from cloud storage.

And that 50ms is processing pages sequentially, using a single core on my little laptop. Plenty of scope to parallelise if I wanted to.

The reason why I'm still using Python, to borrow a slogan from Bruce Eckel long ago, "Python fits [my] brain".

View on HN

50ms doesn't sound especially fast. If you had a million of them to process thats 4 hours, maybe an hour with some parallelism? A 100x speedup from going native then becomes quite welcome. Python has its place, but the fact you even think the tens of ms domain is "fast" or even fast enough on modern CPUs shows the real strength of Python, which is most people dont actually care at all about performance. Thats not to say its performant though. Just that no one cares anymore. Some rando boss is happy to wait 4 hours for your script to chug through a million statements, because no one actually told them it can take 2.5 minutes in a native language. If they did maybe the boss would suddenly speak about their dream to not only process the data but have a near real-time BI panel instead of a batch report so they can react within one business day. The issue with Python is missed value, not the value it can deliver.

Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds.

You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement.

Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total?

Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days.

How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?

One dimension to consider here is cost of compute.

Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size.

At scale, that could translate into hundreds of thousands of dollars in savings.

But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.

> The issue with Python is missed value, not the value it can deliver.

I'm very aware of the business value here! The limiting factor in these types of "messy real-world data" problems is the developer time to get 100+ templates right on all the different variations encountered in the wild. I can iterate on each template extremely effectively in a Jupyter notebook REPL, and immediately rerun a sample of 100 statements for that bank in a few seconds.

While the total corpus of statements I have access to is actually around a million, no one cares how quick processing them all are if the extraction isn't reliable enough!

Exactly. The time spent developing a solution for your problem in pretty much any other language is going to cost magnitudes more than processing millions of documents in python. And that’s ignoring the ongoing maintenance where the hackability of teasing data out of PDFs in ipython is going to top any other system.

I have a soft spot for the work you’re doing since I’ve spent a good portion of my life now extracting data from PDFs and can appreciate the joys of the process more than most.

> If you had a million of them to process thats 4 hours

If you had 20 bank accounts sending you monthly statements for 100 years you'd have 24,000 of them. 50ms each would take you 20 minutes to process the 100 year backlog.

If you have a million of them to process then you're a bank or similar type of institution that can devote resources to this, either computational or developer time to optimize things. A 128 core c6a.32xlarge would turn the runtime from 4 hours into 1.9 minutes and cost $5 to run for an hour.

A laptop generates about 27 grams of CO2 per hour. So going from 4 hours to 2.5 minutes would save about 0.1 kg of CO2 each time the app is ran. So for a company of 1,000 people working 48 weeks per year that's almost 5 tons of CO2 that could be saved.

Ah well, 5 tons of CO2 is a fair trade off for being able to use Python, right?

I understand where you're coming from, and I do care about the climate as much as the next person, but this comparison seems downright silly. Basically, for your example to work every single person have to run the same script every week and wait for 4 hours. Are there even that many bank statements? And then, if you, your spouse, and a child take a single flight coast to coast, round trip, that is 4.5 tons right there.

I agree with your reasoning and your insight about missed value is valuable, but this is not a Python problem. It's a scaling problem.

You can solve scaling problems in many different ways with different costs. Any reasonably sized company has a Kubernetes cluster, just launch 10 instances and you're down to 24 minutes instead of 4 hours. You could also buy faster hardware. You might increase single-node performance by rewriting in a performant language (this won't help much if I/O is the bottleneck though, which often is the case). Talking to your data supplier to check if they can send the data in a machine-readable format is probably also something to consider. Or you might conclude that any effort is not worthwhile at all. Who knows?

Scaling problems are not be avoided as a negative thing, they're beautiful and ought to be celebrated. What's not beautiful is letting the thought of potential scaling problems lead you to make decisions that optimize for long-term results instead of short-terms results.

My problem with Python, especially when reading someone else's code, or my own old bad code, is that I find that Python doesn't fit [my] brain.

The lack of typing (in my experience you always end up with a bunch of libraries that don't have type hints) means I spend most of my time just figuring out what I'm supposed to pass to a function (I hate the django docs. Give me something like docs.rs where I can quickly scan what functions something implements, and see what they accept and what they return instead of forcing me to read an 800 page novel that STILL doesn't give you that basic info), and writing tests and trying to guard against passing the wrong thing around.

In my own code, I always use typehints and write good docstrings (nearly) everywhere.

Visual Studio Code's type checker saves so much time and has improved my code quality to no end. It is especially powerful on polymorphic inputs, making sure the different code paths operate on the input type you expect.

PyCharm is good for this too.

I've recently started using Type Hints by default. At the very least, it makes your code more readable. Nothing bad comes from that.

I started enforcing type hinting in my personal projects. 99% of the time it's easy and useful to add. When I can't figure out how to make a mypy error go away within 30 seconds, I just disable it for that line. I feel this makes for very good 80-20 value/effort tradeoff.

I will say that more than one bug has been found after I disabled the single line hint though! Always surprising to me how powerful types can be.

I do not see the benefit of type hints. Good docstrings (and naming of the function and arguments) are superior, at least for the human. Type hints are too much clutter.

There’s a certain program size from which they start to make sense and further still the benefits are obvious - you have a computer to help solve puzzles for you, so why not use it if the puzzle is ‘will this peg fit into that hole?’ when you have thousands of pegs and holes.

And so you end up writing climb_the_wooden_ladder(l: WoodenLadder) plus climb_the_aluminium_ladder(l: AluminiumLadder) or press all into an OO hierarchy instead of

climb(thing): '''Climb a thing with steps''' # prevent run time crash: if not hasattr(thing, 'getSteps'): hitDeveloper()

No, you define a Ladder (or Climbable if you’re feeling fancy today) Protocol and all remains good in the world.

No need for OO hierarchy, just use a protocol

Oh, I missed the Protocol class of the typing module. Have to further investigate the usage of this.

Thanks to you and baq!

I have a fear that Python will be improved to the point it offers no more advantage over other languages.

> type hints are too much clutter.

Any sufficiently detailed python docstring contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a type annotation.

Which suits duck typing better than type hints.

Docstrings are great until you change your mind about something fundamental.

Once you make that big scary change you might have to rerun your unit tests dozens of times before you've found and fixed all of the bugs that your change caused.

A static type checker, plus a bit of foresight and diligence, will give you a much more complete view of the carnage up front, so that you have a good idea about the cost of making that change while it's still cheap to decide differently.

And the best part is that with python you can only bother with the type hints where you need them (e.g. at the boundaries between teams)

Never experienced that. Maybe you should make your code more modular.

I agree with you there. The experience I was thinking of here involved taking a codebase which wasn't modular enough and making it modular. So the "big scary change" involved changes to the module boundaries (which are the only thing I bother with type hints for anyhow).

Docstrings won't tell you if your code is valid or not. Type hints with proper linting will. Shorter feedback loops = increased productivity.

Since I'm programming in Python, I experienced invalid code only in very few cases, more or less just where the function expected an iterable and I gave a string instead of a list of strings (typing errors are well prevented by the colour hints e.g. neovim gives with the right plugin).

And some libraries sould be better documented.

But the compile time error messages of C or C++ or Delphi most of the time do not deliver a shorter feedback loop, because they are generally ugly and hard to understand. And you should write tests in all languages, so static typing isn't a timesaver in that respect, too.

> The lack of typing

It's gotten a lot better with type hints and typeshed, but I agree, not knowing what type an arg should be and hoping duck typing works out isn't great.

>Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement.

I'm interested in management of domestic documents and similar institutional correspondence, and I'd love to hear more about how you approached and solved your problems with the PDF bank statements.

I have lots of bank statements ... and other documents like utility bills, rates invoices, etc ... and I'd love to just be able to feed them to a script and automatically tease out salient details.

Plenty of languages have a huge package ecosystem. Your problem is that you don’t know them. I’m not saying the distinction is helpful for your situation, but your problem is “I have too much time invested in learning Pythons ecosystem” and not “other languages don’t have all the packages I need.” For REPL that’s another mindset. You use a REPL to experiment and prototype, I use unit tests. You use python because you already learned it. Your brain has been fit to python, not the other way around. You can fit it to other languages if you choose. I am currently learning my 7th “use in production” language.

You're partly right. However Python's sweet spot for me is when writing code to process complex, messy, poorly understood data. Using a Jupyter notebook lets me keep partially processed inputs as live objects, examine and change their in-memory representations, and quickly prototype the next stage of processing. It feels like sculpting with playdoh.

Once that's working, the code moves from the Jupyter notebook into a proper code module with unit tests. Then the emphasis is on building the production system rather than manipulating the data.

I get what you mean with unit tests. Just for me it feels further away from the data, which is where most the complexity in my business domain lies.

Sure. I do use Jupyter in a decent IDE for some adhoc data diving or for a spike prototype before “downcoding” into a cluster.

I have Python analytics capturing HTTP Headers, processing them, and finally storing them in less than 8ms on a single core dev instance. Python is plenty fast for many things.

Also, I love the syntax. It reads like a sentence.

> in less than 8ms on a single core dev instance. Python is plenty fast for many things.

What’s that, nowadays? 25 million cycles, 75 million instructions?

Python isn’t plenty fast, modern hardware is.

More precisely, modern CPUs are so much faster than network or file I/O that if your program has to wait on I/O, the CPU is going to sit idle most of the time anyway, so trying to optimize CPU time is pointless. The GP's use case is an example: he's capturing HTTP headers, so the bottleneck is network I/O, not CPU. So Python for his use case is, as he says, plenty fast.

For use cases where CPU is the bottleneck, such as numeric simulations, Python users reach for something like numpy, where all the CPU intensive stuff is being done in C extensions, not Python bytecode.

The person you’re replying to is saying that Python meets their needs. Judging by what they’ve said about their use case, I doubt it’s a meaningful affirm to their overall pipeline.

Your counterpoint is that it’d be slower if it were running on a Pentium 4. Do you honestly think that they don’t know this?

Yes, this is precisely the point. Things are at a point where Python’s benefits are worth more than what another language would bring in speed improvements. I imagine that there would be less of a benefit to busses if we had to Flintstone them everywhere instead using a combustion engine. That’s not the current reality though.

That sounds insanely slow.

I think it's fine to like Python as a scripting language, but arguing that Python is fast is simply laughable.

I forsee Python's reputation going the way of PHP's (< 5, not modern), and statements like this will definitely contribute to it.

Let's have a contest then. Pick a (relatively simple) web product and your "fastest" language. I bet I'll build it faster with pure Python.

There's no need for a dick measuring contest, there are plenty of life-like benchmarks available online that show Python is agonizingly slow.

> I bet I'll build it faster with pure Python.

What? We're talking about runtime speed, not development time...

You said: "That sounds insanely slow."

Anti-up.

When building a large system, the choice of language is always non-binary. You do not have to commit to one and only one language for the entire system. Python offers a great stating point for most cases.

When you scale, certain modules will start becoming the bottlenecks. At this point one can just re-write the given module as a sidecar and Python by design is easy to bind external libraries.

How are you matching a file to a template like that?

Reply on news.ycombinator.com