Ask HN: Any event sourcing systems that use source files for persistence? | Hacker News Reader

Ask HN: Any event sourcing systems that use source files for persistence?

I've been thinking about event sourcing while also trying to whittle down the number of languages in play in my stack.

It occurred to me that if you're doing event sourcing, you could just persist actual source code that replays the log, i.e. just dump something like this to the filesystem and keep appending to it:

    createUser("rbg@scotus.gov", "dj85jig")
    createUser("knope@nps.gov", "b85kee6")
    befriend("dj85jig", "b85kee6")

It seems like people usually persist to a traditional database, which gets you acces to that whole ecosystem of tools. But source files seem like a natural fit for event sourcing and I'm curious if there are examples of this approach in use? Cautionary tales?

View on HN

Is the aim to do event sourcing for later data analysis or to actually base your app's business logic off of the events?

I have some experience with using events to construct the final state for data analysis. (If you didn't mean for data analysis, just ignore this comment) The problem with trying to reconstruct data at some final state based on events is that the meaning of the events change over time. For instance, createUser might do one thing in January, then something slightly different in April (e.g. sets a default value to be 10 instead of 5). It becomes a nightmare to try to get the final table to be accurate, and if you try, your reconstruction script ends up with a bunch of if/then cases based on the history of the program generating the events.

You still probably want to log the events, but it looks like you are hoping to recreate the users table based on your createUsers history, which would run into the problem I mentioned above.

If you do log data straight to files, you can always analyze it with something like Splunk or Kibana (and if you use Kibana, you probably just log the events with the ELK stack). Still, it's way easier to analyze data with a relational database because of the tools built around SQL, and it's already familiar to analysts whereas Splunk's proprietary language is something they'd have to learn.

If you end up using SQL, I suggest https://aws.amazon.com/redshift/ for the data warehouse, and http://DataDuckETL.com/ for getting the data into Redshift.

We've done something similar where we log requests along with parameters to a sql table. One thing to beware is to make sure you're not logging passwords, credit card info, ssns or other types of pii.

Reply on news.ycombinator.com