Hacker News Reader

I cannot upvote this enough.

One of the experiences I had was applying it to provisioning with idempotent shell scripts, combined with a trivial tool like https://github.com/lloeki/apply

Creating a VM in DO would add the basic SSH keys, then just run the thing against the new IP. Boom, VM ready. Want to make existing VMs config up to date? Apply. Boom.

The whole thing ridiculously scaled up in a totally unexpected way, saving hundreds of hours of headaches and mistakes.

Why not ansible/puppet? Check out the rationale in the readme.

This is basically like going from imperative to declarative.

Describe the final end/desired state and let the machine figure out how to get there, progressing through the work until it's caught up even across restarts or other boundaries.

The outcome is the same. Except that it is not declarative, because 'you' are the machine who codes up that logic.

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc.

This was common when I was writing puppet. It was easy to have your entire puppet run take minutes - and want to run every few minutes - while consuming a LOT of cpu time including on a centralised server and not just the end host.

There are of course ways and strategies to this and not every situational falls afoul of it. But it is something you very much need to be aware of.

I otherwise whole heartedly endorse this.

> If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it.

Definitely! And on top of that, you probably want to apply some business logic to what you're doing.

To expand on the example of "Report should run at midnight":

If you run the check regularly, you can constrain the date range you're checking to just since the last time you checked, and store this date either in memory or persisted depending on your use case, technology, and how you want system failures/restarts handled.

Similarly, if the system was down for several days, it might not be relevant to send all missing reports, but instead just the last one.

On the other hand, if "daily reports" are required for auditing purposes, it probably does make sense to send all of them. But you have to be careful so on the very first run it doesn't go "Hey, there are 18,726 missing reports since Jan 1, 1970!"

It's also important to write the report itself to be idempotent. You can't write it so it reports to "current" time with the assumption it runs at exactly midnight. Instead you have to actually constrain the dates to exactly what you want. Even aside from the idempotent invocation I'm talking about, as your system gets big you'll get thousands of reports that all need to run at "midnight" and that just isn't practical: maybe they run all sequentially, and some will not actually execute until several minutes after midnight.

suppose you serve clients who want their reports to be generated at 02:30 local time. you could try to run a cron container for every time zone :bleh:, but with daylight savings, the 2:30 event can fire twice, or not at all!

instead you simply see if there's a report for yesterday, and if there isn't make one.

the edge cases are still there, see replies

Curious to hear what ways and strategies you considered and landed on.

Is what you are describing not what they call "event-sourcing" ?

Just keep a log of all the events that have already happened (crucial - I have seen "event logs" that were more like "request logs") and to find out the current state of the system - if the notification or report has has been sent - by examining this log.

The lists you are describing kind of sound like an event log of sorts.

It seems to me that this distinction between primary and secondary approach is not really neccessary.

The more structured way to do this is a CQRS framework with Event Sourcing as the underlying data store model. With that you create specialized readmodels that react to events and you can query them for the exact thing you need to know. Once you do something based on that query it will result in new events that update the readmodel. It's a powerful way of designing a system.

Apparently if you want spare cash, little books of this stuff could sell for $5-10. Just became aware myself, setting up to write again.

That’s to say, this comment probably just saved or made me money. I’ve some hands on but clearly not as much as you exhibit. Thanks.

Target audience: Sr SE, reporting to next level up and perhaps offering tradesoffs, or advising peers or reports working on those systems and need a refresher to give a complete answer to a question.

How would you market those? It's been a very long time since I bought a book on software.

If this worked as a business model then everyone would be doing it. Everyone also includes people who don’t actually know this stuff, but want to make a quick buck, giving out bad advice. And if you don’t already have the experience, then you don’t have the knowledge to weed out bad advice.

The idempotent way is definitely better, but I argue/add that it would be better to have the incoming business requirement phrased appropriately such that the idempotent approach is the first one that comes to mind.

I am not sure how to perform that rewording/transformation in general though or even for the examples above.

well today i just learned something very important, thanks!

How can I find more wisdom like this?

I wish I didn't have to be so meme'ey, but to answer your question;

By reading more Hacker News ;-)

Stub your toe a few times until enlightenment forces its way into your field of view.

Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cannot exist. Because our reality is not realtime (or our perception of it is always delayed because our brains/awareness is slow) - we can never truly capture the present moment. Data we see is always in the past. Reality is eventually consistent (from our perspective). All things in our reality runs concurrently, stateless and in their own little time bubble. You can also read up on the concepts behind Event Sourcing.

Anyway, it is a huge rabbit hole, but reading up on the ideas behind Actor Model might just blow your mind. I'm convinced it will make a comeback because it is the natural way our reality works and Actor Model mimics it the closest, and it will also work better for multi-core cpu's. For non-tech reading, try The Power of Now by Eckhart Tolle. Then try to reflect on (relational databases) + (actor model) + (event sourcing) + (power of now) + (brain in a vat philosophy) + (try to to figure out at what speed the universe "renders", how fast our brains can think, how much delay there is, what limits do our body's sensors limits impose) + (try to figure out how closely Object Oriented programming can mimic reality (hint: all programming are tiny simulations that try to mimic some tiny part of our reality)) + (read up on the history of time keeping and the huge rabbit hole of calendars, and how most programming languages attempts to handle time (badly for the most part, it's a legit rabbit hole on it's own)).

After a year or two of hard reading, reflection and teachings (and wrecking your head), you may have good insight into the nature of programming and the fallacies of trying to build realtime systems, or trying to mimic our reality in a computer in general. It is insanely complex, we as programmers merely touch a grain of sand of the beach of complexity. We have no idea how simple our most convoluted systems are in the face of nature, yet we convince ourselves that we KNOW how things work and that we are in control at all times.

Good luck!

Thank you, this will blow my mind. I will take this path. Real time systems is the field which is where I want to be.

To add to this part: "My main epiphany was that realtime applications do not exist. Rather they cannot exist."

Most of us think our applications are realtime (never mind the OS schduling cpu time for your code, so inherently not realtime) but this is only because it behaves as expected, while there is not too much load nor too much data. The moment you start having significant computational load or start having too much data (aka the cpu, memory or storage starts choking), or when you start to split data apart (aka distributed applications) then the problems with "realtime" applications starts to become obvious.

If something goes wrong, to fix the immediate problems would be to retry/rerun the simulation, but that might result in duplicating your intents, which results in the simulation and its state as invalid. Then you either (hopefully) have some mechanism to de-duplicate whatever happened more than once or you have to start you simulation from scratch after fixing the data manually.

Or just design the core of the system with Idempotency in-mind from the start so you have a good way to handle such scenarios. Embrace that most things can/should be handled in a eventually-consistent way. Very few things in life has to happen RIGHT NOW (good luck with that).

An interesting spin on realtime systems are computer games, where we refresh a view of the simulation every few milliseconds (aka 60fps). Every frame gets rendered based on the inputs (keyboard/mouse)from just-after the previous frame. So in effect, the current frame being drawn is just a reaction based on previous inputs. So let me ask you this: in a game engine like this, where would you assign the value of "now"? Right after the previous frame? When capturing user input? When rendering the new frame starts? When showing the new frame but before capturing the new inputs? Where is reality's "now" moment in a computer game? Let's ignore the delays between keyboard/mouse input and drawing from CPU/GPU to the physical screen and lets assume this is a single player FPS and not a multiplayer turn-based game. And lets ignore sound processing too - just think of the simplest of game loops - should we consider them as realtime? Ponder that.

Going a little bit further, there's the time between the frame displaying on the screen and it reaching your retina, then the time between that and it reaching your brain in the form of electro-chemical signals, then the time between that and your brain decoding it.

By the time you perceive anything, and even more so by the time you react to it, it's already in the past.

There is no "now", it's all an approximation.

> An interesting spin on realtime systems are computer games (...) Ponder that.

My younger self did that, quite a lot, thank you. I never felt comfortable with various approach of intermixing input handling, physics/game logic and rendering - whether to run them sequentially, decouple some or all of them, whether the physics should be "look-ahead" or "look-behind"... I ended up picking the pattern that leads to the most stable and deterministic behavior (look-behind physics in lockstep with input and on fixed update rate, rendering independent and done variable-rate), and stopped thinking about it.

So thanks for mentioning it in this context, it makes me more comfortable to hear from someone else that "now" in games is ill-defined on philosophical level, and I shouldn't have worried about it as much as I did.

> "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes.

Ha! You are biased toward the word "Idempotent way". Above - you have totally different steps under "Idempotent way". So, your process is different for each of them. There's no tech differentiator and it should be - as that's how you implement "Idempotent way"

I've set up a bunch of Windows batch scripts similarly, but didn't know the name for it. It's beautiful because failures, crashes, bugs, restarts, etc. barely matter. The next time the system is able to do what it was supposed to do, it does.