itamarst

Yeah, I was using DFA.

Next step is trying alternative approach, but if that alternative doesn't work I'm going to see about wrapping your package for Python.

Thanks for all your work on it!

There's definitely a conversion cost. For strings, Python apparently caches the UTF-8 encoded string, so if you _repeatedly_ transfer it to Rust I suspect (but haven't checked) that the cost is much lower.

In general I suspect it's the usual "NumPy arrays are fast, everything else you better be getting a sufficiently large boost from the low-level code to justify conversion".

For the thing I prototyped in Rust, it was wrapping the `ahocorasick` crate which was in fact faster than `pyahocorasick` which is written in C or Cython or something. Both have similar conversion costs, probably, so it came down to "for lots of data the Rust version was faster".

I've been playing with PyO3 for prototyping, and wrapped some Rust code to see if it's faster than Python. The experience was very much like using Boost Python (whcih these days has alternative with https://github.com/pybind/pybind11). It's _really_ easy to wrap code for Python, and it has nice APIs to ensure GIL is held. Being Rust, I'm much more confident I won't suffer from memory unsafety issues which my C++ at the time did.

Now I'm starting to use it as part of the Python memory profiler I'm working on (https://pythonspeed.com/fil), in this case to call in to the low-level Python C API which PyO3 includes bindings for in addition to its high-level API. This kind of usage is more like writing C, except with the benefit of having high-level APIs (for GIL holding, but also object conversion) available when I need it.

So basically you get safe, high-level, easy-to-use APIs, with fallback to low-level unsafe APIs if you need them.

Highly recommend trying it out.

I do some consulting where I'm paid to fix specific bugs / add specific features to various open source projects, often in Python data science world. Usually it goes something like:

1. Figure out what the project's contribution guidelines are.

2. Check out code, figure out how to build it and run tests.

3. If it's a complex task, might want to build some credibility with maintainers first, so might contribute tiny fix first.

4. If it's a complex task, try to engage maintainers to see if they'll accept it, and to get hints on how to approach it. They may be hard to reach, so this might take a few days or a couple of weeks, though I'm not working during that time, it's just delay.

5. Do the actual work.

6. Go through whatever code review process. Sometimes I might need to gently ask for reviews if it sits for too long.

$80 total for all the above is laughable. And to be fair I ask for a good amount of money because I am good at what I do, and there are people who live in much cheaper places than Boston area and could charge less.

For $80, you'd have to both have an extremely low cost of living and already be an expert on the project for this to be in any way worthwhile.

> "We blocked the phrase and other variations of it from trending," a Twitter spokesperson said. "We want trends to promote healthy discussions on Twitter. This means that at times, we may prevent certain content from trending. As per our Help Center, there are rules for trends — if we identify accounts that violate these rules, we’ll take enforcement action."

E.g. https://www.theguardian.com/us-news/2021/jan/10/hang-mike-pe...

It's been used since the time of Jane Austen (by Jane Austen, in fact), it's perfectly normal: https://pemberley.com/janeinfo/austheir.html

In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com/articles/estimating-memory-usage/

I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.

Hey, been meaning to reach out.

There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely."

The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale.

This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it?

I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil, so would love to talk about it if you're interested.

I can't imagine that has any significant impact on memory usage, at least for normal multi-GB-RAM servers: the docstrings will only be loaded once.

Memory usage is an interesting problem, because the failure mode is so much more painful (crashes and complete lock-ups) than high CPU usage.

And even if you return memory to the OS, that doesn't actually solve too-high memory usage.

Some things you can do:

The article mentions `__slots__` for reducing object memory use, and other approaches include just having fewer objects: for example, a dict of lists uses far less memory than a list of dicts with repeating fields. And you can also in many cases switch to a dataframe with Pandas, saving even more memory (https://pythonspeed.com/articles/python-object-memory/ covers all of those).

For numeric data, a NumPy array gets rid of the per-integer overhead for Python, so a Python list of numbers use way more memory than an equivalent NumPy array (https://pythonspeed.com/articles/python-integers-memory/).

It's a tradeoff, yes. You can do more work to rebuild a base image, or you can say "technically have slightly newer version of glibc isn't reproducible but in practice I don't expect that to break anything so I'll live with the risk".

1. "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end.

2. Given you do want system package updates, you need to deal with Docker caching breaking your security updates. So you need to rebuild your image from scratch, without caching, either every time there is a security update or on a weekly basis. https://pythonspeed.com/articles/docker-cache-insecure-image...

3. Not mentioned: don't use build secrets via ARG.

Some approaches to secure build secrets:

1. BuildKit supports them: https://docs.docker.com/develop/develop-images/build_enhance...

2. Via the network, which is a hack, but it works: https://pythonspeed.com/articles/docker-build-secrets/

3. Via multi-stage builds, but this destroys caching.

Yes. But the Alpine C library is musl, not glibc, and musl can sometimes be incompatible with code compiled against glibc, so I don't like using it.

As other people have said, you can negotiate for this. I've done it at multiple jobs. Here's an interview with someone who has been doing it for 15 years: https://codewithoutrules.com/2018/01/08/part-time-programmer...

1. It's easier at existing job; you have all this knowledge that's hard to replace (https://codewithoutrules.com/2019/01/25/4-day-workweek-easy-...).

2. At new jobs, apply normally. Then _after_ you get an offer, ask for shorter hours.

3. I wrote a book about the process; it's no longer public linked on my site because pandemic has lowered negotiation leverage a lot and I'm not sure how to address that, but if you're interested: https://codewithoutrules.com/3dayweekend/

Debian Stable currently ships Python 3.7. If that's what you want, great.

But if you want Python 3.8, the soon-to-be-released Python 3.9, or other versions, you don't have that.

There isn't too much of a downside, except that you need to be a little more careful in your CI, otherwise you can end up rebuilding from scratch each time, thus losing the benefit of faster builds.

See here for why and how to fix it: https://pythonspeed.com/articles/faster-multi-stage-builds/

Beyond the other two answers, I added this bit because I keep seeing people make this mistake in their StackOverflow questions about why their Dockerfile isn't working.

For image size, it doesn't matter. It does matter for build time.

Specifically, multi-stage builds let you get better caching and therefore faster rebuilds, since you can cache the pre-installed gcc etc. layer, while still getting the small image.

So if you have a human being waiting on frequent Docker build results, yes, multi-stage is better.

In this case, the builds are automated, no one waits for them, so it doesn't really matter (except for burning some extra CPU cycles).

That's not what I meant, but rather "Python is a Conda package too, a dependency." I'll rephrase to make it clearer.

Why do you still have a contract with ICE?

April unemployment is probably only measured mid-March, so it predates all these claims.

I found one of the emails (https://lists.opensource.org/pipermail/license-discuss_lists..., at the bottom)

I would ban someone for that. "Toxic loonytoon" is not a productive way to have a discussion.

(Originally quoted it here but getting formatting right here is terrible).

E-bikes! They're amazing. Bicycle with small assist engine.

You start pedaling, and the bike engine helps you, so going up hills is easy.

Riding one feels like you have super-strength.

The benchmarks were run on a computer with hyperthreading disabled.

I don't mean the filesystem implementation is inherently slow, necessarily: I'm talking about bandwidth. If you get 3× compression, read times from S3 will be 3× as fast—but that means no mmap().

There are two ways of doing this that I know of:

1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance.

2. Syncing to filesystem with AWS DataSync. This would work, yes, but it's a S3 specific feature and not all blob storage systems have it.

h5py at least doesn't support zstandard. The basic compression algorithms built-in to HDF5 don't include ZStandard.

pytables might make it possible though, haven't looked.

I'm the original author. If the reader might get the wrong impression then I need to fix the text, that's my fault not theirs. So I'll do that.

My impression from the talk linked in the article was that HDF5 doesn't do BLOSC by default, and that in general it's much easier to plug new things in, add caching, etc..

Dask, while great, doesn't solve this problem at all. You still need some way to load the data.

And for that Dask supports Zarr, and HDF5, and pre-existing arrays (where mmap() comes in), and a few other formats: https://docs.dask.org/en/latest/array-creation.html#