Stupid question. What was wrong with hot reloading the configuration file? It doesn’t drop requests, doesn’t seem to impact CPU and RAM by much, and it’s straightforward to script around.
But think about a system which is dynamically adding domains with TLS and SNI, or constantly adjusting backend workers. Or trying to automate a clean rolling upgrade.
You want to quickly shed load, or add new TLS certificates, etc. If you have a process which is accepting these change config requests and controlling a file, it potentially has to worry about atomicity if several different things are happening at the same time — could one change blow away another if you script a “read config, update config, link new config, trigger reload, poll till complete” function without locks, one change could blow away another?
Presumably the API would protect against this and allow a simpler function to push a given state change?
But I agree that it’s likely over designed and not strictly necessary assuming the hot reload doesn’t have unanticipated side effects. E.g. What happens to stick tables, session state, etc.?
It’s not like the REST API eliminates all these concerns too. For example, if you make a change via the API, presumably it is persisted across reloads, or there is at least a way to make it be?
HTTP offers none of this. No atomicity and no consistency of the full configuration. Want to edit a service or maybe alter some hosts? Better hope it's already setup as expected before editing and you're doing all the right calls in the right order and they all succeed. You will have to write the hell of a state machine to ensure to get to the expected state.
The only sane use case for live-changes is enabling/disabling a server, which is necessary to perform rolling maintenance or blue-green deployment.
If you're balancing traffic by session count, the new process doesn't know about the sessions in the old process (not sure if you can workaround this, it's not a problem I have, just something I'm aware of)
If you use source port ranges, the new process doesn't know what the old process is still using, so you get extra logs about ports in use, and extra CPU spent on syscalls to find free ports. For me, on a busy system, this makes it even busier for quite a few minutes before it slows down to normal again. I haven't looked at this API yet though to know if it will help for me.