itamarst

https://groups.io is good for mailing lists, yes.

Unfortunately NumPy's thread pool is only for BLAS, the underlying library for numpy.linalg functions, mostly. Other operations are single-threaded. So you need your own thread pool (or process pool) if you want to parallelize anything else.

Yes, but... it's harder than one might think to avoid threads. Merely importing NumPy starts threads, for example.

If I add a `if __name__ == '__main__':` I get the same result.

Here's the gdb stack trace:

  #0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:57
  #1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:87
  #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x17bc7c0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, 
    private=<optimized out>) at ./nptl/futex-internal.c:139
  #3  0x00007ff326c9cc5f in do_futex_wait (sem=sem@entry=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
  #4  0x00007ff326c9ccf8 in __new_sem_wait_slow64 (sem=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
  #5  0x00007ff326c9cd71 in __new_sem_wait (sem=<optimized out>) at ./nptl/sem_wait.c:42
  #6  0x000000000042765b in PyThread_acquire_lock_timed (lock=0x17bc7c0, microseconds=-1, intr_flag=0) at ../Python/thread_pthread.h:483
  #7  0x0000000000625813 in _enter_buffered_busy (self=0x7ff326f330f0) at ../Modules/_io/bufferedio.c:281
  #8  0x000000000045f9b3 in buffered_flush (self=0x7ff326f330f0, args=<optimized out>) at ../Modules/_io/bufferedio.c:825
  #9  0x0000000000524185 in method_vectorcall_NOARGS (func=func@entry=<method_descriptor at remote 0x7ff326f611d0>, args=args@entry=0x7fffa8bc7a38, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at ../Objects/descrobject.c:436
  #10 0x000000000053bd5a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffa8bc7a38, callable=<method_descriptor at remote 0x7ff326f611d0>, 
    tstate=0x17bf410) at ../Include/cpython/abstract.h:118
  #11 PyObject_VectorcallMethod (name=<optimized out>, args=0x7fffa8bc7a38, nargsf=<optimized out>, kwnames=0x0) at ../Objects/call.c:828
  #12 0x000000000060283a in _PyObject_CallMethodIdNoArgs (name=0x8f6120 <PyId_flush.lto_priv.2>, self=<optimized out>) at ../Include/cpython/abstract.h:243
  #13 _io_TextIOWrapper_flush_impl (self=0x7ff326f40040) at ../Modules/_io/textio.c:3038
  #14 _io_TextIOWrapper_flush (self=0x7ff326f40040, _unused_ignored=<optimized out>) at ../Modules/_io/clinic/textio.c.h:685

See the article for an explanation of why this is a bad idea. Note that on macOS Python disabled fork()-based multiprocessing as the default long ago (Python 3.8?) because it's so broken, and it will stop being the default for multiprocessing on Linux in 3.14, deprecated in 3.12.

From memory, the reason it deadlocks is:

1. Writing to stderr grabs a lock.

2. Part of the multiprocessing code (perhaps not present in Python 2) also grabs this lock.

3. If you fork at the right moment (which is quite likely with the loop) the lock is held by a thread that is now dead, and so now you're waiting for a lock to release that will never be released.

I fundamentally don't trust the ADL, so not going to waste time reading their stuff. Based on my political opinions they would claim I'm an anti-semite, and again, I'm Jewish, grew up in Israel, still have family there.

For a much more nuanced view re ADL's claims (from a source you may find more trustworthy than me) I recommend reading https://forward.com/news/521941/jewish-campus-antisemitism-g...:

--------------------

''' The first reports from George Washington University last October were alarming: Someone had broken into a fraternity house and desecrated a Torah — tearing the scroll and dousing it with laundry detergent. The story spread like wildfire online; hundreds joined a campus vigil against antisemitism.

“The status quo, where Jewish students at this campus are fearing for their lives, has existed for far too long,” declared the fraternity president, Chris Osborne, choking up as he spoke.

Many Jewish students found the assertion exaggerated. The incident was upsetting, but hardly life-threatening: The “Torah” in question turned out to be a small replica, the kind you can buy for $20 at a synagogue gift shop.

And whoever entered the house the night before Halloween damaged the Torah while throwing hot sauce all over the kitchen, ripping out smoke detectors and toppling appliances; Tau Kappa Epsilon is not a Jewish fraternity and the motive of the crime was unclear.

D.C. police closed their investigation a week later without any leads. But in certain quarters, the incident was held up as a kind of Kristallnacht on fraternity row, heralding the peril of Jews at universities nationwide. The head of the Anti-Defamation League condemned it as a “blatant act of antisemitism,” and Algemeiner, a right-leaning Jewish news outlet, called it one of the year’s “most shocking instances of campus antisemitism.”

...

Even those who were sympathetic to the missions of national Jewish organizations said the attention they lavish on the campus can sometimes miss the mark. Too often, they said, the spotlight bends toward histrionics, obscures positive Jewish experiences or simply overwhelms the 18-to-22-year-olds as they seek to navigate the complex politics of Israel and Jewish identity while still making it to class on time.

“All these organizations have their own agenda,” said Jessica Carr, 21, a senior and president of GW for Israel. “Even though we might agree with them — we’re all pro-Israel — they all have their own agenda and it’s not necessarily what’s best for GW.”

Beyond college campuses, rising antisemitism in the United States has included synagogue shootings, a rabbi taken hostage during Shabbat services, and beatings of visibly Orthodox Jews in New York. The campus incidents documented by the ADL almost never involve violence and are usually related to Israel. '''

The ADL tries to redefine any support for Palestinians as anti-semitism... they're not a credible source of information on the topic. Nor is the Jerusalem Post on this topic, nor is the NY Post on... anything?

Looking at the Wikipedia article for Washington Examiner, it does not seem credible either.

(Context: I am American Jew who is Israeli citizen and grew up in Israel.)

These AI systems are like a spell checker that hallucinates new words: did you mean to type "gnorkler"?

At least Google (when not using the summarization "feature") doesn't invent new stuff on its own.

This is an excellent summary, but it glosses over part of the problem (perhaps because the author has an obvious, and often quite good solution, namely DuckDB).

The implicit problem is that even if the dataset fits in memory, the software processing that data often uses more RAM than the machine has. And unlike using too much CPU, which just slows you down, using too much memory means your process is either dead or so slow it may as well be. It's _really easy_ to use way too much memory with e.g. Pandas. And there's three ways to approach this:

* As mentioned in the article, throw more money at the problem with cloud VMs. This gets expensive at scale, and can be a pain, and (unless you pursue the next two solutions) is in some sense a workaround.

* Better data processing tools: Use a smart enough tool that it can use efficient query planning and streaming algorithms to limit data usage. There's DuckDB, obviously, and Polars; here's a writeup I did showing how Polars uses much less memory than Pandas for the same query: https://pythonspeed.com/articles/polars-memory-pandas/

* Better visibility/observability: Make it easier to actually see where memory usage is coming from, so that the problems can be fixed. It's often very difficult to get good visibility here, partially because the tooling for performance and memory is often biased towards web apps, that have different requirements than data processing. In particular, the bottleneck is _peak_ memory, which requires a particular kind of memory profiling.

In the Python world, relevant memory profilers are pretty new. The most popular open source one at this point is Memray (https://bloomberg.github.io/memray/), but I also maintain Fil (https://pythonspeed.com/fil/). Both can give you visibility into sources of memory usage that was previous painfully difficult to get. On the commercial side, I'm working on https://sciagraph.com, which does memory and also performance profiling for Python data processing applications, and is designed to support running in development but also in production.

If you're writing a compiled extension, you can use Rust. Or Cython. Or C. Or C++. Which should you use?

TL;DR:

* If you're wrapping existing C library, I'd use Cython.

* If you're wrapping existing C++ library, I'd use PyBind11 (no personal experience, but it's based on Boost::Python, which I have happily used). Cython in theory does C++ but it's a frustrating, limited experience.

* If you're writing a tiny library and you don't know Rust, and you're not worried about memory safety, Cython is nice.

* For anything involving writing extensive new low-level code, Rust with PyO3. Memory safety _will_ bite you in the ass. Concurrency is vastly easier with Rust. You get a package manager for dependencies. You're not writing a pile of code in a language without good tooling (Cython).

Long form, with more alternatives and use cases: https://pythonspeed.com/articles/rust-cython-python-extensio...

Quick overview of the design space:

* PyPy JITs everything, so it can do _normal_ Python numerical code that is quite fast and regular Python code that is fast. However, its interactions with libraries like NumPy add overhead, and it seems like it can't JIT code that interacts with NumPy in a useful way (AFAIK, would be happy to be proven wrong). So not useful for optimizing numeric functions that interact with libraries like NumPy.

* Plain old NumPy and friends. This is great... if the operation you want is already available as a "vectorized" API. "Vectorized" in this context does NOT mean SIMD, it's a Python-specific usage, see below.

* Numba: JIT compilation specifically focusing on interop with NumPy and similar libraries. Lets you write subset of Python but unlike NumPy you can use for loops and go fast.

* AOT compilation: Cython, Rust, C++, etc.. You have a longer feedback loop, but you have a full programming language, especially if you avoid Cython. OTOH Cython has nicer Python interop so for simple just-a-little-addon it can be easier to use if you don't already know Rust. You really shouldn't be writing new C++ in this day and age (but wrapping an existing library is useful). Like C++, Cython doesn't help with memory safety. Cython also suffers from two compilers, so debugging can be harder, especially if you use the C++ interop; if you are wrapping existing C++ library, I'd probably start with PyBind11 based on long-ago experience with Boost::Python.

Longer form:

* "Vectorization" in the context of Python: https://pythonspeed.com/articles/vectorization-python/

* PyPy and Numba as alternatives to vectorization: https://pythonspeed.com/articles/vectorization-python-altern...

* Choosing a compiled language: https://pythonspeed.com/articles/rust-cython-python-extensio...

* The performance overhead of AOT compiled libraries (less relevant if you're doing anything numeric): https://pythonspeed.com/articles/python-extension-performanc...

* Numba intro: https://pythonspeed.com/articles/numba-faster-python/

You can buy chest freezers that have conversion to refrigerator as a built-in feature, e.g. https://www.bestbuy.com/site/amana-16-cu-ft-chest-freezer-wi...

(Appears common enough that BestBuy has it as a search flag).

Yeah this was a copy/paste mistake, I have fixed it.

Have you tried PyPy?

They're shipping PGO builds of the Rust compiler, so for faster compilation you don't have to do anything ("Windows builds now use profile-guided optimization, providing 10-20% improvements to compiler performance", per Rust's release notes: https://github.com/rust-lang/rust/blob/master/RELEASES.md).

For pile-of-strings data, there are still things you can do. E.g. in Pandas, if there are a small number of different values, switch to categoricals (https://pythonspeed.com/articles/pandas-load-less-data/ item 3). And there's a new column type for strings that uses less memory (https://pythonspeed.com/articles/pandas-string-dtype-memory/).

It takes like 5 minutes, and once you are in the habit it's something you do automatically as you write the code and so it doesn't actually cost you extra time.

Efficient representation should be something you build into your data model, it will save you time in the long run.

(Also if you have 100s of columns you're hopefully already benefiting from something like NumPy or Arrow or whatever, so you're already doing better than you could be... )

There's just a huge amount of waste in many cases which is very easy to fix. For example, if we have a list of fractions (0.0-1.0):

* Python list of N Python floats: 32×N bytes (approximate, the Python float is 24 bytes + 8-byte pointer for each item in the list)

* NumPy array of N double floats: 8×N bytes

* Hey, we don't need that much precision, let's use 32-bit floats in NumPy: 4×N

* Actually, values of 0-100 are good enough, let's just use uint8 in NumPy and divide by 100 if necessary to get the fraction: N bytes

And now we're down to 3% of original memory usage, and quite possibly with no meaningful impact on the application.

(See e.g. https://pythonspeed.com/articles/python-integers-memory/ and https://pythonspeed.com/articles/pandas-reduce-memory-lossy/ for longer prose versions that approximate the above.)

Does anyone actually use cache-oblivious data structure in practice? Not, like, "yes I know there's a cache I will write a datastructure for that", that's common, but specifically cache-oblivious data structures?

People mention them a lot but I've never heard anyone say they actually used them.

I'm on vacation, will do Monday.

Fil can dump memory profiling reports on failed allocations; as other commenter said, Linux by default happily just gives you memory even if doesn't have any, so failed allocation implies giant allocation.

https://pythonspeed.com/fil/docs/oom.html

BTW I am starting a Slack for devs working on profilers, would be great to have you all join, I'd love to hear more about the ELF patching technique (Fil uses LD_PRELOAD and macOS equivalent).

Cool! Fil also tracks every allocation too, although it doesn't dump that at the moment, just the resulting report.

I'm excited to see more profiling tools for Python!

This sounds like it does peak memory, which is critical for batch jobs, since that's the bottleneck. Memory is fundamentally different than performance in that it's a limited resource, instead of cumulative cost; making any part of the program faster almost always helps speed up the program (at least a little, or at least reduces CPU load), but optimizing non-peak memory has no impact. You have to be able to identify the peak in order to reduce memory usage.

If you want peak memory profiling for Python that also runs on macOS, check out https://pythonspeed.com/fil/ (ARM support has some issues, but once I unpack my new Mac Mini I plan to fix it.)

Ways memray is better than Fil:

- Native callstacks.

- More kinds of reports, and ability to do custom post-processing of data.

- Much lower overhead (but not always, see reply).

- Subprocess support.

Fil I suspect has better flamegraphs: https://pythonspeed.com/articles/a-better-flamegraph/

And if you're running Python batch jobs, and want both peak memory and performance profiling in production, check out Sciagraph: https://pythonspeed.com/sciagraph/

(You can probably cobble together something like Sciagraph with py-spy + memray, but you won't e.g. get timeline reports designed with batch jobs in mind.)

Note that line needs to be omitted from the time measurement, since it's the same for both scripts and is just overhead, I was just measuring the actual mean + substraction code.

I reran a few more times, got a bunch of variability, but did get some runs where f64 is just twice as slow, not 10×. Probably my initial run hit swapping. So I will update the article.

That being said, worth noting that this is also quite hardware dependent. Like, I just have 16MB L3 cache, I imagine higher L3 would change ratios a lot.

UPDATE: Oops. Original runs were with NumPy 1.18, this run was with NumPy 1.22. So will rerun again with that.

UPDATE2: NumPy version doesn't matter. So yeah was probably swapping.

UPDATE3: Fixed article should be up in a minute.

It's not stable ("continuous delivery", so packages are changing out from under you), so it's probably not a good basis for a Docker image.

The difference, of course, is that the engineers and CEO won't end up making $200 that week: https://www.businessinsider.com/doordash-delivery-driver-in-...

A better approach is to avoid writing bash scripts. They're just too dangerous, even with "strict mode" (http://redsymbol.net/articles/unofficial-bash-strict-mode/).

Better to use Python, or Ruby, or a Rust or Go executable.

MA sequences 30% of PCR tests, and has so far found only a single Omicron case...