Hacker News Reader

Comment by the__alchemist | Hacker News Reader

That's because you're doing web stuff. (I/O limited). So much of our computing experience has been degraded due to this mindset applied more broadly. Despite a steady improvement in hardware, my computing experiences have been stagnating and degraded in terms of latency, responsiveness etc.

I'm not going to even go into the comp chem simulations I've been running, or that about 1/3 the stuff I do is embedded.

I do still use python for web dev, partly because as you say, it's not CPU-bound, and partly because Python's Django framework is amazing. But I have switched to rust for everything else.

View on HN

As a java backend dev mainly working on web services, I wanted to like python, but I have found it really hard to work on a large python project because the auto complete just does not work as well as something like java.

Maybe it is just due to not being as familiar with how to properly setup a python project, but every time I have had to do something in a django or fast api project it is a mess of missing types.

How do you handle that with modern python? Or is it just a limitation of the language itself?

That's 100% an IDE thing. I use Zed (or Emacs or anything else supporting an LSP) and autocomplete is fast and accurate.

Pycharm has been fine. Just disable the AI stuff and you get accurate completion. It even has completion for Django ORM stuff, which is heavily dynamic.

Pycharm has "AI stuff" now?

I won’t completely argue against that, and I’ve also adopted Rust for smaller or faster work. Still, I contend that a freaking enormous portion of computing workloads are IO bound to the point that even Python’s speed is Good Enough in an Amdahl’s Law kind of way.

It's been said that Python's greatest superpower is that it's the second-best language at the most stuff.

No one's really developed an ecosystem for a language that's more performant that can match it, and that's all it needs to assert dominance.

I've never understood this. Python cannot be optimized like C, C++ or Rust. It cannot do advanced functional things like OCaml, Haskell or Scala. It cannot run in browsers like TypeScript. It cannot do games programming like C# and it can't do crazy macro stuff like Clojure. I don't think it's even second best at those things.

I'm reading this as, "It cannot do things the best", and that's correct. It can't.

But it can do them well enough, and enough people know it that they can drag a solution across the line in most domains.

I hear this a lot, but can you really say that you're consistently saturating a 1Gbps line for netcode or 6+ Gbps nvme for disk data? In my experience this doesn't really happen with code that isn't intentionally designed to minimize unnecessary work.

A lot of slow parsing tends to get grouped in with io, and this is where python can be most limiting.

I don't personally use Python directly for super IO intensive work. In my common use cases, that's nearly always waiting for a database to return or for a remote network API to respond. In my own work, I'm saturating neither disk nor network. My code often finds itself waiting for some other process to do that stuff on its behalf.

> That's because you're doing web stuff.

I guess you didn't notice where he talked about running numpy?

And 300ms for a DB call is slow, in any case. We really shouldn't accept that as normal cost of doing business. 300ms is only acceptable if we are doing scrypt type of things.

> in any case.

In some cases. Are looking up a single indexed row in a small K-V table? Yep, slow. Are you generating reports on the last 6 years of sales, grouped by division within larger companies? That might be pretty fast.

I'm not sure why you'd even generalize that so overly broadly.

To put in perspective, 300ms is about looping over 30GiB data from RAM, loading 800MiB data from SSD, or doing 1TFLOPS on a single core computer.

300ms to generate a report would be able to go through ~100M rows at least (on a single core).

And the implicit assumption that comment I made earlier, of course is not about the 100M rows scan. If there is a confusion, I am sorry.

That's all true, so long as you completely ignore doing any processing on the data, like evaluating the rows and selectively appending some of them into a data structure, then sorting and serializing the results, let alone optimizing the query plan for the state of the system at that moment and deciding whether it makes more sense to hit the indexes or just slurp in the whole table given that N other queries are also executing right now, or mapping a series of IO queries to their exact address in the underlying disks, and performing the parity checks as you read the data off the RAID and combine it into a single, coherent stream of not-block-aligned tuples.

There's a metric boatload of abstractions between sending a UTF-8 query string over the packet-switched network and receiving back a list of results. 300ms suddenly starts looking like a smaller window than it originally appears.

There is nothing for us to take away in this discussion. So let me be the first to tune down: all I want to say is: don't take that 300ms as given, it sits in this uncomfortable region too short to be an async op and too long to be noticeable (anything between 50ms and 2s fits this bill). Most likely the query is doing something suspicious and would benefit the most to take a closer look at.

I was totally with you until that last sentence, then you lost me again.

Saying a DB query is too long by giving an arbitrary number is like saying a rope is too long. That’s solely dependent on what you’re doing with it. It’s literally impossible to say that X is too long unless you know what it’s used for.

Reply on news.ycombinator.com