How does the plugin system work? I thought Go didn’t allow for dynamically loading code, where you able to find a work around?
The main OctoSQL process then connects to that gRPC server for each plugin.
Omitting a bunch of details, each record stream will end up being a gRPC streaming connection.
This approach is cross-platform (unix sockets work on Windows since recently!) and more than fast enough (millions of records per second), while being easy to keep organized (everything stays on the file system).
That said, all of this is abstracted away by the plugin SDK, so you only have to implement the right interface and the rest will get handled by the SDK.
Here's an example plugin that simple and fairly easy to read: https://github.com/cube2222/octosql-plugin-random_data
And here's the gRPC schema for plugins, which might be useful to get a better picture: https://github.com/cube2222/octosql/blob/main/plugins/intern...