On Building Glue Systems
devpoga.org
devpoga.org
For a long-time we've heard the opinion that complex systems are best written in a combination of a "systems" language (say C++ or Java) and a "scripting" language (say Tcl, Lua or Clojure.) That's how video games are written, for instance.
Clojure and Elixir have the particular angle of being suitable for concurrency, which more conventional scripting languages (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various global interpreter locks.
- convert the data output/need as input in core data structures - Don't use classes or custom types to encapsulate data and make it accessible using a custom interface - instead manipulate the data structures instead using the large standard library. - convert the data back into the required format for storage/output/etc.
Very ETL-like, that is needed as different systems that need to be glued together may be written in different languages and most likely will export serialised data or write in a database. Certainly binary formats are rare nowadays.
Abstractions like the one for sequences allow reuse of the same function for different data structures (you don't have to convert things into lists for processing). It's possible to have other, non-core data structures implement the same abstractions, so that the existing library can be used.
ORMs are not desirable with that approach, for example. The query result tables are usually converted into things like Clojure hash-maps instead being created into unique objects with different ways of accessing the encapsulated data.
Then you have functional programming, which while nice by itself also allows using your cores efficiently. Less of a need for additional "infrastructure" glue that way. Asynchrony is also provided using libraries like core.async or agents.
For what it's worth, each instance of the new "Ractor" concurrency primitive in Ruby 3 has its own interpreter lock: https://docs.ruby-lang.org/en/master/doc/ractor_md.html
Does this really matter for a glue language in a distributed system? Async for non-blocking IO and then just run multiple processes. Allows for things like killing a process and letting the OS cleanup. For example, in Java you're not supposed to just kill a thread.
I've written a lot of "glue" in Python using asyncio, websockets, AQMP and similar things for my IoT system at home. The coding is fun and the performance seems excellent, but that's because the workload is low. If I was at the point where I needed more than one core worth of performance I wouldn't be so happy.
One case where I am a little frustrated with Python now is a "batch job" that processes and resizes images. There are numerous half-baked ways to farm out the work to separate processes. These would be "good enough" but they are still half baked, and the frameworks that give a little structure to the process have one thing in common: they don't run on Windows.
In a pure "glue" case there is also an Amdahl's law kind of situation: it's not crazy to have a 32-core or more system today and a single-threaded "glue" system that dispatches tasks will have a hard time keeping those cores busy if it is spending even 2% of the CPU effort that it takes to do the real work.
One memorable situation was sitting in a sales meeting in ~2003, showing the speed of one such system. We let the potential client customize some input data, then ran a computation (the point was to show that something that used to be a batch process could now be interactive). It took ~30 seconds on our laptop running on battery power. On the spot, their CTO accused us of lying because it took days of dedicated mainframe time for them to run the same computation. Our sales person later asked if we could introduce configurable delays that we'd only turn on during sales presentations so that it would be easier to believe and less threatening. Our original implementation was 1000s of lines of Python and would take weeks to run. Over time we optimized that down to about half a day of run time. Then we figured out that the thing taking all the time was equivalent to finding shortest paths in graphs over and over again. So we found an open source C extension with a nicely optimized implementation of Dijkstra's algorithm and our Python code reduced to 100s of lines and was literally unbelievably fast.
Today I'm working on a simulation system for use by people who are experts in game theory. The thing being simulated is an algorithm in an online system (that almost everyone reading this in the US has used) that does server side optimization of the selection and placement of content during web / mobile app page loads. We want to share code between online and simulation to avoid semantic differences. The online system is in Python. Python is what our math folks know. The algorithm currently runs tens of millions of times per day online with billions of inputs, all on a small cluster of machines. Online we need low latency, O(1 ms) per run. Offline we need as much computational bandwidth as we can get to keep math folks going - improvements to this algorithm can mean big revenue wins. We haven't shipped yet, but the current version replays hundreds of millions of runs (10s of billions of inputs) per minute in a large Spark cluster (developers only ever see Python, not the PySpark API). The online backend runs the exact same Python code in O(100 µs) with a Python backend (only one non-builtin extension is used online - Pyrsistent, which gives us efficient immutable data structures - the online backend uses FP concepts to avoid complexity).
In my experience, running up against things like the GIL for doing large amounts of computation means I'm stuck in a local optimum for speed. Backing up and thinking about my architecture / algorithms gives me a real chance of making improvements that show that the GIL wasn't my core issue. Eliminating the GIL might get me an order of magnitude speed improvement (often at the expense of more complicated code). Finding better architecture / algorithms often gets me several orders of magnitude of speed improvement and simplifies my code at the same time. A very high level language that is good at gluing things together gives me more time / options for doing that.
1) "Complex systems should be designed like video games"
2) "Complex systems should be designed around a huge event bus SaaS"
Video games/engines are architected around responsiveness (low latency) & content creation.
Sending every little event to a remote SaaS for processing is sure to add tremendous latency.
To be fair, many video games due make heavy use of event processing/message passing. But the architecture of a video game event system & SaaS event bus are quite different for performance, security, reliability reasons.
Complex distributed systems are, well, complex, and can't be reduced to a simple formula or architecture. Better to spend time accurately understanding the problem that is being solved.
I think "glue systems" can be generalized by saying: systems that consolidate events, and react to events in real-time. Most everything that happens in your system is an event, and things need to react to this. For example:
- A glue system will almost certainly plumb stripe events info a billing system, plus other systems for things like failed payments.
This system shouldn't go down (HA). It should be able to react to changes easily - as biz ops change frequently. It should be auditable, easy to debug, and easy to deploy.
The first pass at this in startups is usually... just doing stuff in the API request, or building webhook endpoints that manage this for you within the API. Then, you might add Sidekiq, or SQS/Lambda. Then, you might build out event-driven architecture using SQS/Kafka. It's kind of a pain, and still it's not easy to grok, debug, or architect.
Inngest handles all of this for you, though. It consolidates events from every system (internal and external, via APIs, webhooks, oauth, etc.) and then allows you to run DAG-based workflows in real-time whenever events are received. With full logging, debugging (step-over debugging), retries, user-auditing, changelogs, version control, etc.
I think that event-driven systems and glue-based systems overall haven't had much love in the developer UX, but I'm hopeful we can change that. If you're interested in using us or working with us on building the platform, ping me :)
Once that's done and I've refactored some stuff I'm planning on opening up the core runner. Our FoundationDB persistence we're working on may likely remain closed - we will provide an interface which allows you to persist stuff and a psql adapter :)
That'll likely happen early next year.
Sounds like modern web development.
For example, there are plenty of tools to create ML model endpoints (send data, receive prediction) but many times, those models are not customer/user facing. They will be hit by some other program/system (often on a regularly timed schedule).
One solution is to embed the model in the application that needs the predictions, but this can get hairy when it comes to updates/maintanence/etc.
Over the next few years, we will see some generalized "model invocation" components that integrate with common data sources for these use cases.
You can see an inkling of this with the seldon-core kafka integration: https://docs.seldon.io/projects/seldon-core/en/stable/stream...
I think a glue system is meant to connect disparate pieces that otherwise could not interact with each other.
I think of shell as the language for glue (and we need a better one). and I explored this relationship in a followup to a popular post [1]: