I used to work at ITA, and no, it's not semi-scraped, it's doing same calculations airline reservation system does to approve a fare.
5,611 karma · joined February 24, 2016
I write about fundamental engineering skills and programmer career advice at https://codewithoutrules.com
I also write a weekly email about all the mistakes I've made both coding and in my career over the past 20 years, so that you can learn and avoid them: https://softwareclown.com
At its bottom layer, Dask takes a graph of tasks and dispatches them to a scheduler. Schedulers can range from a few threads, a few processes (using multiprocessing), a few process on local machine (with the Distributed scheduler), or tens of thousands of machines.
That latter example is not made up, I've seen geoscience demos doing that in 3 lines of code on one of their giant clusters.
The task graph can be generic Python functions, using e.g. the bag API, which gives you a map-reduce-y like API: https://docs.dask.org/en/latest/bag.html
Separate, and optionally, Dask also has emulation layers for NumPy, Pandas, and scikit-learn I believe as separate project. Here you write code that looks like normal Pandas code, say, but instead of executing immediately it creates a execution graph, and then you submit the graph to your scheduler and it runs it for you. These emulation APIs are partial, by their nature, but also optional.
One interesting use case for this emulation API is processing larger-than-memory datasets. In ordrer to support multiple processes, and even more so multiple machines, Dask reimplements the NumPy and Pandas APIs using batching underneath. And so you can run a single machine and take advantage of that batching to process data that wouldn't fit in memory when using normal Pandas APIs, while also getting access (optionally) to multiple CPUs: https://pythonspeed.com/articles/faster-pandas-dask/
(If you specifically want Pandas APIs, there are other alternatives mentioned here in there in comments, e.g. Modin.)
My main personal experience was using the lower-level API to run image processing code in parallel, on a single machine with multiple worker processes. It worked great.
One approach is RSS, the memory in physical RAM... but what if you're swapping? Then again, maybe you swapped out memory you don't actually need and ignoring swap is fine.
The other approach is "how much memory you allocated", and then you hit fun issues mentioned in this article, like "the OS doesn't actually _really_ allocate until you touch the page".
(Longer version: https://pythonspeed.com/articles/measuring-memory-python/)
After we'd hired him, we asked about the old company again. "Oh, it was an utter shitshow" and he went on a long and very compelling rant about how badly managed it was... in far more detail than he had when interviewing.
I feel like this is a reasonable way to do it; don't badmouth your current employer while job searching, but no need to stick around unnecessarily.
1. What rustc does with crater (linked from article), where you can actually test real code. On a smaller scale, the author of the article points out cryptography package for Python, which runs tests of its most popular upstream dependencies. This latter is probably feasible for many libraries.
2. Analytics, though this requires work to make it private and anonymous, and for libraries maybe requires system-level infrastructure. (Again, this was suggested by Alex.)
3. Beta testing, surveys, and other forms of user feedback.
1. Stay at current, unpleasant job. 2. Take a huge paycut.
But there are other jobs. So what you do is, you start applying elsewhere, and meanwhile you try to negotiate with the startup.
The chances are that they won't go up that much. That's OK, you can:
1. Stall them, and use existing offer for negotiating with another company that gives you an offer. Even just mentioning you have another offer is helpful.
2. Negotiate for something else. "Hey, this is 50% less than my current salary. Would you be willing to raise it by 20% and then have me work 4 days a week?" I haven't done the math to figure out if this makes sense, but there might be some combination of hours and higher offer that might work for you. Most startups are unlikely to say yes, but I have successfully worked shorter work weeks at startups.
A few things the article talks about where one can maybe do even better:
1. likely()/unlikely() not being in stable Rust. This is true, but the hashbrown project has some hacked-up variants it claims work on stable: https://github.com/rust-lang/hashbrown/blob/bbda6e0077bafb75...
2. Rust not having fast thread locals. Same problem for me, so likewise did it in C with "initial-exec". But! If you use clang, you can get LTO across C and Rust, so you can get fast thread locals _and_ not have function call overhead. Basically need to use same version of Clang as Rust does (12 at the moment) and do a little song and dance in linker and compiler flags. See https://matklad.github.io/2020/10/03/fast-thread-locals-in-r...
3. For testing these sort of things, being able to assert "this test touched this code path" is extremely useful. In my case, for example, I have different code paths for sampled and unsampled allocations, but from perspective of code calling malloc() everything should be identical. So how to tell if correct code path was used? Coverage marks are a great solution for this: https://ferrous-systems.com/blog/coverage-marks/
(The Python profiler, if anyone is interested: I've already released an open source memory profiler that tracks all allocations, https://pythonspeed.com/fil/. Unlike most memory profilers it tells you source-of-allocations at peak memory, which is key for data processing applications. The commercial production-grade version I'm now working on uses sampling, and will be even more focused on data processing batch applications; the goal is to have essentially no performance overhead so it can always be on.)
If your goal is to find peak memory usage, which is what you care about in data processing batch processes, Fil is probably a better choice (https://pythonspeed.com/fil).
https://pythonspeed.com/articles/security-updates-in-docker/ has an example of a security update that was missing in the official Ubuntu base image, two weeks after Ubuntu released the updated package. I've also seen missing security updates in the Debian base images used for things like Python and OpenJDK official base images.
This means that you need to rebuild automatically yourself on a schedule (or as result of distro updates), e.g. https://pythonspeed.com/articles/docker-cache-insecure-image...
2. Only scan for security problems that _have updates available_. If there are no updates ... there's nothing you can do. Many security problems will never get updates, because it's an obscure problem that e.g. only happens if you install a NFS server in your container (which I assume you're not), or the upstream maintainer has closed it as WONTFIX. Some scanners (e.g. Trivy) have this as a command-line option, or a switch you can toggle in the UI.
Long version of the latter: https://pythonspeed.com/articles/docker-security-scanner/
Long version: https://pythonspeed.com/articles/docker-security-scanner/
Second, the author isn't installing security updates! You need to run `apt-get upgrade`, even for official images.
Long version: https://pythonspeed.com/articles/security-updates-in-docker/
Third, Alpine has some issues in some cases, although for Go at least neither of these issues is usually relevant so Alpine is fine.
1. musl can be subtly incompatible with some applications, with annoying bugs. Personal experience: if you using minikube in a WeWork office, Alpine-based (or really, musl-based) containers would fail to resolve DNS inside Kubernetes due to a concatenation of circumstances that was mostly the fault of WeWork's ops team but which glibc handled better than musl. The problem has since been fixed by WeWork, AFAIK.
2. For Python specifically, binary precompiled packages (wheels) won't work on Alpine, which means you ahve to recompile the whole universe, which means container builds are slow. There's a PEP which might get this fixed, but for now, not worth it. https://pythonspeed.com/articles/alpine-docker-python/
I do hope there's a new edition eventually, but as he says the core is still the same.
https://journals.aom.org/doi/abs/10.5465/1556350
Hawthorne effect... was also made up.
> But only five workers took part in the study, Dr. Ross said, and two were replaced partway through for gross insubordination and low output. > > A psychology professor at the University of Michigan, Dr. Richard Nisbett calls the Hawthorne effect ''a glorified anecdote.''
(https://www.nytimes.com/1998/12/06/weekinreview/scientific-m...)
I learned both these facts from the highly entertaining "The Management Myth": https://mwstewart.com/books/the-management-myth/
If you'd ever like to chat about the automation parts, I'd be interested in hearing how you're approaching it; the niche of "scientific computing, but repeatable" is quite different than traditional scientific software, and it seems like people are still in early stages of figuring out how to do it.
Would also be interested in hearing how you approach correctness. The best approach I've discovered is metamorphic testing. Basically you modify real inputs, and then ensure the output matches. E.g. you say, "OK, I have this finished algorithm that segments cells, `f(image) -> cells`. Now, if I double brightness on everything, I would expect the same results, so let's see what `f(brighter_image)` is, I would expect same output." Or like "If I merge nuclei-looking splotches that cross what original segmentation boundary was, that should result in fewer cells." Unfortunately only discovered this technique after I left the image processing job, so haven't had chance to try it.
* When Asia Was the World
* Religions of the Silk Road: Premodern Patterns of Globalization
1. For server workloads, the main issue is usually leaks. Lacking leaks you can just characterize the workload, give it enough memory (usually it's not very much per request), and call it a day.
2. For batch data processing, the main issue is ... using lots of memory to process the data :) Like, loading 4GB of data and doing stuff to it can easily use 20GB of RAM if you're not careful.
For the latter case, and specifically for Python, I've written an open source memory profiler (https://pythonspeed.com/fil) that helps you spot which code allocated the memory responsible for peak memory. You can also use it for leaks, but that's not the main use case.
It has some performance overhead, so it's not usable in production, but I'm also working on version that will be fast enough to run in production, at the cost of slightly less accuracy. For batch processing workloads that loss of accuracy isn't meaningful, mostly, who cares if you're off by 1MB on a 20GB peak memory usage. For server workloads with memory leaks... might take a lot longer to catch the problem as a result of reduced accuracy, and a tool designed specifically for server workloads might work better by taking a different implementation approach.
In addition to vmprof, pyinstrument is another alternative.
1. A Docker image is the intersection of literally 50 years of technology, starting from Unix in the 1970s, none of which are perfect. Decisions from back then still have an impact, e.g. you need to get signal handling right if you want shutdown to work (https://hynek.me/articles/docker-signals/).
2. Getting it right is all about _details_. Lots and lots of details, all of which need to be correct. Signal handling, where logs go, security updates, on and on.
3. Every organization does things slightly differently.
So there are multiple approaches:
== Build an abstracted tool ==
A tool will support certain conventions and ways of doing things... but as soon as you want to diverge too much you'll start having issues, because it only supports its particular way of doing things.
Pros: Easy for users.
Cons: Only so long as they don't want something the tool can't handle.
The problem is that it's quite difficult to build a tool that works for _all_ organizations. If you've built a custom buildpack for your company, you've built a custom tool, which a pretty good solution, but still every organization has to build their own tool.
== Build a configuration language ==
In order to support all the edge cases, you build a configuration language... and pretty soon it's super complicated because there are so many different things people want to do. In fact, you end up with something with complexity of Dockerfile. Probably can do better than "it's a shell script", but it's not going to be simple, and you're back to "users have to figure out all the details on their own".
Pros: Flexible.
Cons: You still need to get details right.
== Template ==
Here you do something tool-like: it knows about conventions, is customized for specific common use cases. But, you copy the code into you application repository, so you can then customize anything that doesn't fit.
I have built a template for production Docker packaging of Python applications (https://pythonspeed.com/products/pythoncontainer/) and this is the approach I went with.
Pros: Works most of the time out of the box, can be customized when it doesn't.
Cons: Hard to update if you're ending up doing too many per-application customizations.
---
Obviously this is a simplification, and these options tend to merge at the edges. But fundamentally it's a hard design space and there is no magic bullet, just tradeoffs. Longer version, presented slightly differently: https://pythonspeed.com/articles/developing-tools-for-ops/
$ virtualenv --python=python2.7 /tmp/venv
$ /tmp/venv/bin/pip install --upgrade pip
...
Successfully installed pip-20.3.4
$ /tmp/venv/bin/pip install flask
...
Collecting flask
Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
...
Sometimes maintainers screw this up, but so long as the maintainers do this everything Just Works: people on old Python get old packages, people on new Python get new packages.
E.g. `df["column_name"].values` will you get you a NumPy array.