Time travel in Postgres. Bisect to the last valid transaction | Hacker News Reader

Time travel in Postgres. Bisect to the last valid transaction

neon.tech

73 pointsnikita16 comments

Wow, so it's branching and PITR combined! Seems like a powerful combination of features. PITR is limited to 7 days during the preview.

Yes, in the paid tier we can make it much longer (or a user selection). It puts pressure on the storage, but that's also moves to S3 so the costs are not too bad.

We do have users that created 1Tb of history for a sub 100Gb database.

I'm Neon CEO. Happy to elaborate how this works or any additional questions you might have.

Did I understand correctly that with the "Time" or "LSN" options for branch creation I do not have to create my branches ahead of time (via UI or API), but I can do that _after_ I messed up already?

Correct!

This is a little off-topic, but it's somewhat hard to judge whether I want to try a DBaaS without any indication of pricing. For example, PlanetScale is charging $2.50/GB of storage which quickly adds up.

I know paid tiers are on your roadmap for next quarter, but I don't really want to start building something without some knowledge of what a company is looking to do for costs. Neon is such an exciting project and it's awesome how much you're a part of the PostgreSQL community and open source. Still, it's hard when there's no pricing details.

Hi, I'm on the PM team over here at Neon working on our pricing/ billing. First off, thanks for the appreciation! It's always gratifying to hear folks being excited about what we are building.

As for pricing 1) everything will be consumption based with an option to set guardrails 2) there will be an option to purchase consumption upfront via a "pre pay model" or "pay as you go".

Our goal is to be competitive with the pricing offered by Aurora V2 as we consider Aurora V2 to be one of the standards for serverless databases (even if we disagree with aspects of their approach). The actual numbers are still being worked on as there is further optimization in our product that will help us reduce our unit economics and by extension pass those savings on to our end users.

I understand this is somewhat of a nothing answer but I hope this helps. Is there any other clarification I can add? Also, we are always happy to hear feedback on these decisions. If you want to set up a call or simply share more of your thoughts you can email us at feedback@neon.tech.

It's really cool. I have been following NeonDB for some time. You guys are doing amazing stuff. Just wanted to know how can it quickly create so many branches based on lsn and can run query on top of it? Is it postgres's inherent capability or NeonDB's magic work? I'm excited to know some technical stuff behind this. Please elaborate.

I admit I just skimmed this. But, realistically speaking, what are the DR benefits over just plain out-of-the-box Postgres PITR?

Primarily, the time-to-recovery to an arbitrary point in time is much lower than when you use vanilla solutions, because Neon doesn't need to recover from a previous checkpoint (potentially hours of WAL), nor does it need a file system / data directory snapshot from around the recovery point. pg_rewind can only do so much, and snapshots require planning ahead for the need for recovery.

The relevant bit of the article, if you are not familiar with eg git bisect:

"The next day, you noticed your mistake and discovered that you deleted all users from the table instead of Alice. Unfortunately, you are not sure when exactly the query finished executing and rolling back to an arbitrary date is not an option.

Since Neon offers the ability to create point-in-time branches, you can generate a sequence of branches that include past data in chronological order between two LSNs. You can then go through this list of branches and accurately determine when the disaster occurred."

More generally a lot of recovery scenarios involve investigating to find when and what went wrong, which can be prohibitively hard with slow heavyweight timetravel mechanisms.

Wouldn't really call this time travel since you can still have data loss between branch creations and having no query support for historical data would make it tedious.

We have all the machinery to enable querying the past potentially specifying LSN in the connection string.

What Ux would make sense?

It is possible to create branch retroactively

XTDB: https://xtdb.com/

Love the random xtdb reference here. Truly a unique product that I hope becomes easily usable when xtdb2 comes out with sql-first support.

Reply on news.ycombinator.com