Hacker News Reader

Comment by 8organicbits | Hacker News Reader

    db.commit()
    
    # Give Mycelite time to sync before script exits
    import time
    time.sleep(5)

This really needs a way to wait for the sync to complete. Sleep won't cut it.

View on HN

Relevant HN post from _yesterday_, https://news.ycombinator.com/item?id=36465085

That’s just the example client code, not the sync code. It’s showing you how to demo that the sync works. You’re not expected to use that in production.

https://mycelial.com/docs/get-started/python#observing-synch...

What's the code that you're supposed to use in production to wait for sync then?

I mean to be fair, even in using something like Postgres logical replication, there's no way to verify whether a particular change has replicated (minus xmin hackery).

If I were to write a tutorial on how to set up PG replication and "observe the replication", I think that would probably involve "write to master, sleep for a bit, then see the data is in the secondary."

Couldn't you just query the replication log positions on the primary and standbys and calculate their delta?

I asked this to ChatGPT 4 and it suggested something like this that seems reasonable:

Get it from the main server: SELECT * FROM pg_stat_replication, and the write_ls column has the log transaction location; Then get the location of the replication log and how much has been received: SELECT pg_last_wal_receive_lsn(); Then you can figure out the delta from any of these: SELECT pg_wal_lsn_diff(write_ls, pg_last_wal_receive_lsn()); or similar.

That seems like it isn't a hack.

The problem is that Postgres is a long lived server so the app can fire a commit to the DB and then exit. Here, the app needs to perform the sync, so you can't exit. I didn't see a way to confirm the sync was complete either, so when can you safely exit?

I'm personally setting a 5h sleep here in this thread to wait for a relevant response to this

You wouldn't 'wait for sync', you would just be continually running the application loop, and it would meanwhile be synchronising?

This is just demo code, run these two short scripts to see a synchronising example, at least that's my reading.

I think Mycelite is either very early in it's life, or is just an experiment. The replication code[1] has this at the top:

  //! Replicator prototype
  //!
  //! ** For demo use only!

And the configuration suggests it sends some data to https://us-east-1.mycelial.com [2] And that's not really explained anywhere I can see.

I don't think the authors would currently suggest using it for anything real.

[1] https://github.com/mycelial/mycelite/blob/main/mycelite/src/...

[2] https://github.com/mycelial/mycelite/blob/main/mycelite/src/...

Sync notifications need to be async. That goes for `fsync()`/`fdatasync()` as well as for `COMMIT`. In the case of a database-using application you can report transaction non-failure (as in: it was well-formed and accepted) in a UI separately from transaction durability success (i.e., it really finished), and if you have replication to worry about, you can also report replication success separately.

Every blocking operation needs an async variant. Everything should be async-first.

Fingers crossed there is no network issue. `time.sleep(5)` is ridiculous.

Then just time.sleep(10) bro, it aint rocket science.

I would use time.sleep(60) just in case.

time.sleep(300) allows for a smoke break per sync, please consider my PR

That depends on what you're smoking, let's just be safe: time.sleep(3600)

3600 seconds! That's almost 4 hours!

No it's not; 3600 seconds is 1 hour!

I said "almost"!

Entire field of distributed systems solved by this one trick!

I see so this is a PoC hobby project and nothing to actually use. Thanks for the warning.

Oof

Those silly Internet Protocol guys thought we'd use UDP multicast for such things. That was before everybody knew that "cluster" was just a code word for 5 VMS 1-2 core instances scattered between continents, laptops, and cloud providers.

where did u find this python stuff..

Languages Rust 100.0%

did the link change??

https://mycelial.com/docs/get-started/python#observing-synch...

Link?

https://mycelial.com/docs/get-started/python/#observing-sync...

Reply on news.ycombinator.com