Golang Package: plugin | Hacker News Reader

Golang Package: plugin

tip.golang.org

107 pointspythonist30 comments

I agree with this comment from Reddit four days ago:

> The inability to unload a plugin defeats the use I would have for this, unfortunately. A long running service with plugins may want to load new, updated versions of an existing plugin. Not being able to unload the old version, ends up creating stale references which will keep piling up for the lifetime of the process. As I understand it, this is essentially a memory leak. [...]

> https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco

There is also this project by Hashicorp: https://github.com/hashicorp/go-plugin

Amazing, so this is essentially a high performance alternative to Go scripting. One can already include the go build toolchain in a binary, right? Having JIT is only small steps away.

Disagree with the problem of unloading plugins. From the sizes of Go binaries and memory sizes, one could easily load 10k+ plugins before the process needed a restart. Would be nice to have GC, but having plugins is such an increased benefit. To me this sounds like, Git is crap because you can't unload stored objects.

Nice to see this coming, one step closer to have Oberon System 3 with Gadgets rewritten in Go. :)

Curious what it would take (if at all possible?) to load such a .so from C. Because that would open the door for writing Go libs for all your favorite interpreted languages, web servers, etc.

You can already create shared libraries that expose a C API using -buildmode=shared.

That's already possible, I've written an API that encapsulates Go libraries as Redis modules.

Link?

It's a WIP and I haven't had the time to complete it, but here it is. You can load modules, handle redis commands and return responses, but you can't access redis itself yet.

https://github.com/RedisLabs/rgm/blob/master/example/module....

here's a pretty cool example: https://blog.heroku.com/see_python_see_python_go_go_python_g...

This is awesome, a plugin structure like this is one of the fundamentals of macOS programming (they're called NSBundles in those cases.)

> A plugin is only initialized once, and cannot be closed.

Why not?

I guess it's because if you load a symbol there's no clean way to invalidate it, especially in a thread-safe way. Something like that anyway.

I think it's safe to say there is just plain no way to invalidate it. Even if it were hacked together, it would be unusably dangerous for even trivial cases.

Module loading is something that can be added to a language later, but I suspect safe module unloading is something that has to go in from day one to ever work right. Even in languages that support it (Erlang) it's got a tricky list of caveats to keep track of that strike me as fairly essential to the problem, rather than accidental (in the Fred Brooks sense of the terms).

Because unloading, while possible is a pain to get right.

Hence why Java and .NET restrict it to Class Loaders/AppDomains.

For those who wonder why: some of the things you have to make sure of are if you want to implement this correctly:

- no thread is running code in the plugin.

- no thread has a return address on the stack somewhere in the plugin.

- no variable stores a pointer pointing into the plugin (This includes objects whose destructor is implemented in the plugin)

- no signal handler points into the plugin.

- no kernel thread is reading the plug-ins memory.

In addition there are race conditions to take care of (e.g. thread T unloads a plugin that thread U is loading)

With a 64 bit address space, I guess you can get around most if not all problems by mapping the memory that was mapped to the plugin as "can't read, can't write, can't execute", so that you crash hard if somebody tries to access it, but if you do that, you can just as well keep the code around.

Go is garbage-collected, so it likely already has a lot of the logic to do this, but even then, combining the functionality to get a robust implementation of plugin unloading is a lot of work.

I see an API fail coming. It seems plugin.Load takes the filename WITH extension where it shouldn't. Okay, right now it seems it's only supported on Linux but it ends up in Windows, you'd have to patch code to remove the .so. Mono handles it pretty well, the extension is optional.

I don't think so. I think it is better that the file naming is handled in the application. It is less restrictive, e.g. you can use custom extensions this way.

Consider another instance where file names are automatically mangled: GCC linking with libraries with the switch -lfoo. That has honestly caused me no end of problems. It would have been better if they required an explicit -llibfoo.so from the start.

Go already has several easy methods for OS-specific code if you want.

Not sure it should be be part of this package, but it's trivial to write a function that exposes something like "SharedLibraryFile(libname string) string". Depending on the OS it just formats the default shared object name.

For extra runtime speed you could build OS specific versions of that function using build flags, so for each target OS it has one implementation and you have no switch/case or anything in runtime.

Not necessarily. https://tip.golang.org/src/plugin/plugin_dlopen.go

" 5 // +build linux,cgo"

The Windows version isn't there yet - it could encapsulate that logic.

Assuming or magically guessing file extensions leads to all sorts of confusion attacks that could result in arbitrary code execution.

I would go so far as to say an absolute path should be required.

Didn't one of the main selling points of Go is one big static executable with no external dependencies?

What's next? Generics?

Although this can be abused, the use case is not external dependencies IMO, but rather extending systems. Think of a web server etc. Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime.

This is not like dynamic linking hell you can get yourself into when writing C.

> Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime.

In my Go code, I've just used subprocesses for that. Stdin & stdout, plus a simple serialisation format, and I'm done.

Not suitable for high performance stuff, of course, but good enough for my needs.

yeah, it could work for sure if latency is not an issue. It's also possible to do a go->C->go bridge to load Go plugins dynamically.

But isn't that exactly how dynamic linking works? By loading a DLL at runtime? The only issues I've ever had with dynamic linking are at _runtime_, where XXX.dll can't be found.

yes, my above comment wasn't phrased that well. but do distinguish between a plugin system where you load so's or DLLs to extend an app, and an application that has dynamic linking against them, where the linker handles this.

So basically something like apache modules - either you can load the module or not - vs something like "cannot find symbol foo in foo.so.0.1."

> This is not like dynamic linking hell you can get yourself into when writing C.

No, but they will eventually need to handle plugin versions.

Are you saying that, in cases where that's not possible, you'd rather not have a program at all instead of a program with external dependencies?

Because that doesn't sound very reasonable to me, even after ignoring that plugins aren't really dependencies anyway.

Standard library packages are not the same thing as the language and are not mentioned in the Go specification. The standard library is just a collection of packages that implement common functionalities that might otherwise lead to multiple competing implementations (like hyper, tiny_http, rotor-http, and solicit in Rust). Including a plugin mechanism is not an endorsement of that style of architecture, but simply an acknowledgement that there are programs that will need plugins.

Generics would be great, but they are much trickier. It took Java many years to implement them, while having fewer runtime constraints.

Reply on news.ycombinator.com