Hacker News Reader

Why is "Static linking" bad? A majority of software engineers are shifting to statically linked to reduce runtime complexity.

Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.

It depends on the problem. Someone could say that static linking optimizes for runtime costs but for the problems that OP (and me) are trying to solve, I find that it increases runtime complexity since:

+ It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management).

+ Requires compilation.

+ Licensing woes that do not apply with dynamic linking.

+ Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear).

On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes.

For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.

> The parts I don't like:

> + Static linking

It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm.

> + No extension (shared library) support

> + The module importers look over-engineered / too complicated

Man, the project just came out, give it time. You're very harsh on this.

> + Custom Python distributions

Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking.

> + Not sure what Rust gets you here, besides additional complexity

- link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking

- free rust / python interrop. This is very interesting.

- potentially cross compilation and even make the tools stand alone itself

> + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)

Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.

It's been a while since I looked at this (or at Windows, for that matter), but back in the day one thing that consistently failed for these approaches were thread initialization - after you load a DLL, every new thread gets a call to DllMain with reason DLL_THREAD_ATTACH; You can hook CreateThread and friends, but some threads are not created through those but rather through native APIs.

I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).

https://docs.microsoft.com/en-us/windows/desktop/api/libload...

This tells how to stop it from happening.

But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.

One way is to use the Windows library loader and simply trick it to load a DLL from memory rather than disk.

If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.

How do you trick the loader to load from memory?

There are many ways but the one I chose is in my presentation that I linked above. Roughly, first you get an idea of the system calls involved when a DLL is loaded. Then you hook the system calls that are responsible for mapping/reading in order to redirect them to a memory buffer. Conceptually, it's straightforward. The devil is in the implementation details as they say.

I see. Thanks.

I was under the impression you needed to have some non-trivial tokens to patch NtDll in a way that kernel32 would use (which would make it unusable as a general technique), but it's been ages and I'm probably misremembering.