zknill

presumably a dupe of: https://news.ycombinator.com/item?id=47046068

It seems like this post is a response to the discussion on this previous HN post[0].

There are two main criticisms of "Event [sourcing|driven]" patterns. First, that storing a full history of events becomes a backwards-compatible schema support hellscape nightmare after 12-18months. Second, that the patterns encourage eventual consistency in areas that could really benefit from strong consistency.

This post does a good job of explaining differences between Event Sourcing and Event driven to address problem 1. But does nothing to help problem 2.

Even the event-driven example in the post uses a shopping application where multiple "OrderPlaced" events could be produced without the necessary inventory to fulfill those orders.

I'm just not convinced that most people need this kind of complexity, when a simple database with strong consistency would get a lot of folks a lot of the way there.

[0]: https://news.ycombinator.com/item?id=45962656

I recently had a brief brainstorming with someone building an AI data enrichment tool. I found myself suggesting the 'entities' in the tool would just be semi-structured json blobs; instead of normalising the data into some relational database schema.

I couldn't find good arguments against just slinging the data into a blob and getting AI Models to make sense of it.

This will work, so long as people trust the results are not (too) skewed by paid ads.

I recently used Claude and ChatGPT for exactly one of the examples; comparing different bikes to buy. They could both look up the bike specs and geometry online and tell me what the 1 degree difference in head angle or 5mm difference in reach would feel like to ride. They both did really well.

But I used them only (with cross checks) because I was fairly sure they were giving me unbiased info. As soon as the "discovery" phase of this shopping research becomes polluted with adverts, the product becomes much less useful. The same as "no one trusts online reviews anymore".

I think you might struggle to "scale the read and write sides independently".

It's a real stretch to be describing a postgres view as CQRS

> You can't blame event sourcing for people not doing it correctly, though.

Perhaps not, but you can criticise articles like this that suggest that CQRS will solve many problems for you, without touching on _any_ of its difficulties or downsides, or the mistakes that many people end up making when implementing these systems.

Please explain how you intend to use different models for reading and writing without there being some temporal separation between the two?

Most all CQRS designs have some read view or projection built off consuming the write side.

If this is not the case, and you're just writing your "read models" in the write path; where is the 'S' from CQRS (s for segregation). You wouldn't have a CQRS system here. You'd just be writing read optimised data.

Anyone who's built, run, evolved, and operated any reasonably sized event sourced system will know it's a total nightmare.

Immutable history sounds like a good idea, until you're writing code to support every event schema you ever published. And all the edge cases that inevitably creates.

CQRS sounds good, until you just want to read a value that you know has been written.

Event sourcing probably has some legitimate applications, but I'm convinced the hype around it is predominantly just excellent marketing of an inappropriate technology by folks and companies who host queueing technologies (like Kafka).

It's rare that you come across a product where everything you use works so well for you.

The kagi AI search results triggered with "?" and the Kimi K2 model from assistant are both excellent in helping find what I actually want to see.

Love kagi, keep it up.

I created a side project ~3 years ago based on a similar idea. It was before LLMs were a big thing, and AI could render the code relationships for you.

I started with go and java (the two languages I was using in my job) and built AST parsers that would extract the relationships between the classes and functions and draw them with graphviz. Then I created a filter syntax based on regex to filter the graphs.

I never followed through on the idea to a production ready version, but it helped massively as a personal tool when working on unfamiliar codebases.

The site is still here, but lots of it is probably broken by now..

https://packagemap.co

This reads like a bit of a smell. I'd be pretty suspicious of why you have tasks and reminders in the first place. The question also reads like the volume of these tasks and reminders is large enough to be a problem.

You should check out the "toil" section of the Google SRE book

https://sre.google/sre-book/eliminating-toil/

> If a human operator needs to touch your system during normal operations, you have a bug.

Typically applications will have some kind of logical separation of the data.

Given this is targeted at replication of postgres nodes, perhaps the nodes are deployed across different regions of the globe.

By using active-active replication, all the participating nodes are capable of accepting writes, which simplifies the deployment and querying of postgres (you can read and write to your region-local postgres node).

Now that doesn't mean that all the reads and writes will be on conflicting data. Take the regional example, perhaps the majority of the writes affecting one region's data are made _in that region_. In this case, the region local postgres would be performing all the conflict resolution locally, and sharing the updates with the other nodes.

The reason this simplifies things, is that you can treat all your postgres connections as-if they are just a single postgres. Writes are fast, because they are accepted in the local region, and reads are replicated without you having to have a dedicated read-replica.

Ofc you're still going to have to design around the conflict resolution (i.e. writes for the same data issued against different instances), and the possibility of stale reads as the data is replicated cross-node. But for some applications, this design might be a significant benefit, even with the extra things you need to do.

Looks like it uses Postgres Logical replication to share changes made on one postgres instance to another. Conflict resolution is last-write-wins based on timestamp. Conflicting transactions are logged to a special table (pgactive_conflict_history), so you can see the history, resolve, etc.

https://github.com/aws/pgactive/tree/main/docs

IANAL but, you should be in contact with your airline about your specific flight.

You are entitled to be re-booked on the next available flight or get a refund. If you take a refund the airline has no obligation to you anymore. You might find after taking a refund, the price of an equivalent flight is now much more.

> A spokesperson at MongoDB said: "The rise of MongoDB imitators proves our document model is the industry standard. But bolting an API onto a relational database isn't innovation – it's just kicking the complexity can down the road. These 'modern alternatives' come with a built-in sequel: the inevitable second migration when performance, scale, and flexibility hit a wall."

I think the reason that a there are so many MongoDB wire compatible projects (like this postgres extension from microsoft, and ferretdb) is because people have systems that have MongoDB clients built into their storage layers but don't want to be running on MongoDB anymore, exactly because "performance, scale, and flexibility hit a wall".

If you can change the storage engine, but keep the wire protocol, it makes migrating off Mongo an awful lot cheaper.

The other commenters have mentioned doing deploys behind a proxy, which is fine, but eventually you're going to have to re-deploy the component that terminates the client connections (i.e. websocket server or SSE, etc).

From the client's perspective, there's not a lot of difference between the server dropping the connection (on redeploy) or the connection being dropped for some other transient reason.

That is to say, with a decent client side handling of connection state you just incrementally rollout your new servers and each server terminates its connections triggering reconnects from the clients.

The hardest part is often maintaining continuity on some stream of events. That is; picking up exactly where you were before the connection dropped. You need some mechanism for the client to report the event it last received, and some way to "rewind" back to that point on the stream.

I work on the team that built a Postgres pub/sub connector at Ably (we call it LiveSync).

Our postgres connector also works on LISTEN/NOTIFY and has horizontally scalable consumers that will share the load.

There's two types of LISTEN/NOTIFY you can use. Either have the notify event carry the payload or have the notify event tell you of a change and then query for the data. If you choose the second option you'll get much better resilience under load, as the back pressure is contained in the table rather than dropped on NOTIFY.

If you do go with a poll-on-change design, you'll likely benefit from some performance tuning around denouncing the polls and how big a batch size of records to poll for.

As for the exact collaboration features, when clients are online and interacting, it's fairly easy. Lots of the hard stuff comes from knowing when a connection is dropped or when a client goes away. Detecting this and updating the other clients can be hard.

Another team at Ably worked on that problem, and called it Spaces.

A quirk of Go is that I can cast a `[]string` to `interface{}`, but I cannot cast `[]string` to `[]interface{}`. This blog post is my go-to explanation for why the second is not possible but the former is.

The ElectricSQL concept of 'shapes' is interesting. Right now, a shape is just a single table with some column filters and a where clause. The shape is defined on the fly in the query string. Electric does single table sync of those 'shapes' a lot like Supabase realtime.

I work as part of the team building Ably LiveSync, a competitor in this space. Our postgres connector[0] works in the opposite way by taking rows from an 'outbox' table and fanning them out to subscribed clients over websockets in realtime.

Here's the thing; for any meaningfully complicated data in a relational database the data is likely to be normalised across tables with relations, and there are going to be joins. So I'm really curious how people are making single-table-sync work. (Maybe this is where Electric imagine the future of shapes, solving for joins).

In LiveSync we do it the other way around, instead of having a live-query style subscription to a table (like ElectricSQL), we listen for someone writing a row to the 'outbox' table and that row is automatically sent to subscribers. This means that you're writing your denormlised data directly to the outbox and it's being sent on to clients, rather than writing your normalised data to tables but being limited by single table sync. Or worse, your clients having to subscribe to multiple sync streams and trying and stitch the data back together. We opted to have the write-side insert the denormalised data, rather than having the read side have to stitch normalised data back together.

Electric will at some point try and solve multi table sync, which isn't easy given how the Postgres replication protocol works. It's also not easy to imagine how shapes (which are defined on the fly right now in the query string) will adapt to multiple tables. There's going to be a tradeoff between a complex query string trying to stitch the normalised tables back together, or a 'shape' becoming an entity in Electric which would define how to stitch the normalised tables back together (which you would have to CRUD manage, and update in Electric every time your schema changed).

[0]: https://ably.com/docs/livesync/postgres

By putting all the state into a single CRDT then yes, you're going to get conflict resolution across the entire document. But you might also want to think about how big that CRDT state is going to get.

There's no splitting or isolation available here, you need to load the entire Y.Doc to get any of the content of the page. The content of the Y.Doc is opaque bytes, and if you dump it to some other representation (like JSON) then you lose the internal Y.Doc counters that make conflict resolution work.

I don't know US commutes, or US geography too well. But it seems these two locations are about 45mins-1hr drive from each other.

Where do folks actually live in those areas? Is it that a 30min drive north to San Fran becomes a 30min drive south to San Jose?

Also checkout Ably LiveSync, works with postgres and makes some different (potentially simpler) tradeoffs

It's a server authoritative system, so the final state is determined by the server. There's a single global order of changes (basically the order the changes were processed by the server), so any conflicts are basically resolved by the server, but are applied optimistically on the clients when online or offline.

I can't tell exactly from your question if you're asking about the client going offline before or after making the change, or before or after recieving the server's version of the change, but either way the servers version wins and the client will have to catch up after it's back online.

Various parts of the UK political system looked at a 'social media tax' that would then be funneled into the national health service for mental health.

The idea was scrapped when it became clear the US would retaliate in some way (to protect their big companies).

The article gives one example:

> Today, physical beat-downs are filmed and uploaded to school-based fight groups on Instagram and other platforms—where they live on more or less indefinitely (sometimes they’re forced down, but they inevitably seem to crop back up, according to educators).

If a piece of content is taken down, it should remain down, and it should be the responsibility of the company hosting that content to ensure that it does not reappear.

If the 'cost' of not complying (regulation, fines, etc) is large enough, then there's an incentive to invent cost effective ways of complying with this requirement.

Right now, there's no real benefit to the social media companies to invest in this area. It's too easy for them to say both 'the content is user created and not our responsibility' and 'when we are informed about the content we remove it'.

I'm confused why most of the discussion about these issues is based on blocking the means of access to the content rather than making those hosting the content responsible for it.

Here we're talking about banning smartphones, rather than making the social media companies responsible for their various harms. There's an idiom of 'internalise the externalities', which basically means making the party responsible for a cost bear the cost themselves. It's a useful way of making a market efficient and self-regulate, by making something that costs society also cost the producer.

This kind of narrative seems to be pervasive. For example; we aren't asking the biggest polluters to bear the cost of their pollution (e.g. airlines having to reduce carbon in the atmosphere in line with the amount they produce). We didn't ask the tobacco companies to bear the cost of the public health impact they create.

Instead of blocking or banning the means of access (smartphones), we should be 'internalising the externalities' and making the social media companies bear a cost inline with the negative social cost they create.

Why do you recommend a heartbeat table to mitigate WAL slot growth if the PeerDB Stream targets a specific table? Presumably this means that the WAL slot is subscribed to all table changes, even if only specific tables are actually included in the CDC? Why not just subscribe to the WAL that you need?

Two things:

1. You want to land on the mid-foot. Heel-striking puts a lot of pressure up you leg and joints as you are basically braking with every step. Forefoot strike puts a lot of pressure on your calf and achilles as you have to 'bounce' on every foot-strike to support your body weight. A mid-foot strike is the most efficient transition of energy into and back out of the ground for forward motion. Small steps help you to find a mid-foot strike, large steps (over-striding) will create a heel-strike.

2. For lots of people taking up 'running' they (naturally) believe that they should run. But for many, running continuously will be beyond their zone-2 cardio. It's much easier to start with a jog/walk and build up. In the UK we have a brilliant app called 'couch-to-5k' which is a progressive build from essentially no fitness (walking some distance) up to being able to continuously jog for 5k.

Of course it's possible to solve. But I think the solutions (and the cons of those solutions) are often pretty arduous or unreasonable.

Take the example of joining multiple streams, you have to have all your data outside your database. You have to stream every table you want data from. And worst of all, you don't have any transactional consistency across those joins, it's possible to join when you've only consumed one half of the streaming data (e.g. one of two topics).

The point is, everything seems so simple in the example but these tools often don't scale (simply) to multiple tables in the source db.

All these tools built on Postgres CDC have tutorials that are based on a single table. But in real life, your data models are likely to be normalized across multiple tables with joins. Or you might have an aggregate object that contains multiple child objects from another table.

The hard part of using streaming data as the data source is that you don't have a mechanism to go back and get the data that you missed. Either for some join across normalised tables or to fetch some child objects you've not seen yet.

I don't see how any reasonably complicated data model would work here.