Awesome Python

Honestly, I think a lot of the complaints I have below are not so much against the libraries: I use a lot of the stuff below, often because it is the best available. But I think we too often kid ourselves about the states of these libraries, and ignore the ways in which they don't support us — we can "easily work around them", to the point where we might do so unconsciously, and we stop aspiring for better.

There are some … odd? … selections, e.g., ctypes, curses, logging, venv, pathlib, mimetypes, unittest, unittest.mock — these are just modules in the standard library. I suppose they serve a purpose, and perhaps linking to them by broad topic area allows someone unfamiliar with them a hook back into their own stdlib, but that seems like a Google query could do that too.

pipenv: This, someday, will be great, I hope. But recently releases have been essentially completely broken (which has reduced my trust in it), it doesn't completely implement its own spec for Pipfiles, and it still uses virtualenv when the newer venv would be more appropriate. (I also wish they'd adopted something like Cargo's syntax for version numbers. SemVer is a PITA in it, as you must manually write out the range.) Last, it sometimes fails to find packages matching the requirements in the Pipfile, even when solutions exist.

Pandas: I really think Pandas results in hard to maintain software. The library itself is a maintenance headache: it links to one and only one version of numpy, and will error if it detects a change there. We use a sort of internal PyPI index s.t. we can share binaries, and this causes whoever builds Pandas to also choose what version of Numpy is associated with that version of Pandas for all consumers. Panda's main datatype "Dataframe" is about the most obfuscated type name I have ever hit — what is a dataframe? They share a lot of the same issues as lists-of-dicts: it allows the programmer to not have to define the structure of the data they're working with, which leads to subtly different/shifting/implicit types as data flows through a program, and makes it consequently harder to reason about due to the lack of ontology/naming this creates.

gevent: gevent made some sense in Python 2, before there was a standard async framework. But stuff written in it does not compose: you cannot compose two separate pieces of code that independently work w/ gevent: the combination of monkey patching and cooperative threading means all it takes is one piece of code to block and the whole program can grind to a halt. E.g., we recently hit a bug whereby calling sh (a library that launches and waits for a child process) in one thread caused the entire program to arrest. The "problem" is that sh makes two threads: one calls waitpid, and one interfaces w/ the child's stdin/stdout/stderr. The waitpid call is blocking even in gevent, but what was a separate background thread is now a "greenlet"; the waitpid call won't return until the child gets data on stdin and then runs, but the thread that supplies it is blocked by the waitpid: deadlock. It's not a sh bug (sh doesn't have any say over gevent) but gevent doesn't think it's a bug (waitpid is blocking, not our problem). The end result is that introducing gevent requires continually global reasoning about your program, and that is hard to do correctly.

SaltStack/Ansible/Fabric: Salt has issues around not returning results for commands run (it will time out on nodes, but not really report that). We also had issues w/ Salt falling over in large ish environments, b/c large batches of VMs would overwhelm it w/ traffic, disconnect, and then start attempting to reconnect and form a thundering herd. Like many tools in its league (so, it is not alone here), it believe I am running a set of command across a set of homogeneous instances, and that's that. Unfortunately, my needs are more complex, and sometimes VMs have interdependencies that I cannot express in these systems. (E.g., action A on node N1 depends on action B on node N2. Salt really only allows dependencies on actions within a node. Ansible is similar here too. (Salt has some high level support called "orchestration" IIRC to address this, but the model it uses is fundamentally wrong: I want to just build a dependency graph of action that might run against different sets of nodes, or even cross nodes. Even in homogeneous nodes, sometimes I need to elect a master, e.g., due to initializing a Raft peer-set, and that means the rest of the nodes need to wait on that to happen and then join that new peer-set.) Also, YAML is a poor language for writing what really is a program in. I'd much rather have a real language like Python here. (Unfortunately, especially around how these programs model the problem, I feel like the whole space is mired in one broken set of thinking: that my machines are homogeneous and that I am going to run the same set of commands in parallel on them. It isn't.)

sh: I see this get used way too often, when a `subprocess.run` would be far more appropriate. It has some questionable defaults, like TTY allocation.

boto3: It doesn't really … library. The library basically ends up returning to you what appears to be the raw JSON sent back by the server. It doesn't really wrap the AWS APIs in a way meaningful for Python. (That said, it's the library you use to talk to AWS, unfortunately.)

awscli: Defies a lot of basic command line conventions in ways that are annoying. Is it "help command", "command help" or "--help". Required options bug me, too.

uwsgi: This thing's logs talk about vassals and serfs, the emperor and seppuku, and no, I am not making that up; sorry, this has no place in a production system — logs should be straight, boring, and to the point. It implements a custom binary version of HTTP, which makes introspecting systems written with it hard, as you cannot just curl them. (There is a side util that someone wrote called uwsgi_curl, but last I saw it had issues with Unix sockets, which is basically always the case.) It tries to be supervising daemon and HTTP server all in one, and does too much instead of doing one thing well.

fuckit: I mean, I don't think there is anything wrong with the library. I think it's more a thing of "if you have to use it…".

To have a lifetime of more than a few months, it would need to connect the Pypi and Github API . Even better would be cross-language "ORM-> Django (Python), gorm (Go), Diesel (Rust)" with a way that lets each community hook their package APIs together, categorize the packages, and sort by stars or reviews. They all look similar, but it would need a lot of open source.