HNHacker News
TopNewBestAskShowJobs

p10jkle

393 karma · joined October 23, 2017

submissionscomments
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error)

The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we re-execute your code, fill in any previously stored results, so that it can 'zoom' back to the point where it failed, and continue. It's like a much more efficient and intelligent retry, where the code doesn't have to be idempotent.

p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
not necessarily - we store the intermediary states of your handler, so it can be replayed on infrastructure failures. if the handler changes in what it does, those intermediary states (the 'journal') might no longer match this. the best solution is to route replayed requests to the version of the code that originally executed the request, but: 1. many infra platforms dont allow you to execute previous versions 2. after some duration (maybe just minutes), executing old code is dangerous, eg because of insecure dependencies.
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
I wrote two blog posts on this! It's a really hard problem

https://restate.dev/blog/solving-durable-executions-immutabi...

https://restate.dev/blog/code-that-sleeps-for-a-month/

The key takeaways:

1. Immutable code platforms (like Lambda) make things much more tractable - old code being executable for 'as long as your handlers run' is the property you need. This can also be achieved in Kubernetes with some clever controllers

2. The ability to make delayed RPCs and span time that way allows you to make your handlers very short running, but take action over very long periods. This is much superior to just sleeping over and over in a loop - instead, you do delayed tail calls.

p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
It's not on 1.0 yet, and it was written by someone external (who we have hired, but hasn't started yet :D). So I would say Go is still in coming soon mode, unless you want to just do a POC on 0.8.1 (which is fine, someone on the discord is running that in prod anyhow)

Super excited to hear what you do with it though!

p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
I don't think its a template, I'm afraid! Its a webflow site though
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
Probably one of our two most requested languages. We absolutely are going to do it, probably in the next 6-12 months :)
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
we talk a bit about compensations in the post: https://restate.dev/blog/graceful-cancellations-how-to-keep-... the gist is that you can just use catch statements and put rollback logic in it. Restate guarantees that handlers run to the end, so there's no risk that it somehow won't reach the catch statement due to an infra failure. So catch, rethrow, and then all the way up the stack, the compensations will run
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
Someone already contributed an MVP; in the next few months we'll be adopting it fully and upgrading it to 1.0 (we hired the awesome Azmy after he built it) https://github.com/muhamadazmy/restate-sdk-go
p10jkle··on Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust
Hey all, I work with @sewen, and I focus on the cloud platform which also launched today (https://restate.dev/blog/announcing-restate-cloud-early-acce...) Happy to answer any questions :)
p10jkle··on Restate raises $7M for its lightweight workflows-as-code platform
I've been working on the cloud platform for this for some time - has been a lot of fun. Happy to answer any questions if people have them
p10jkle··on Code that sleeps for a month: Solving durable execution's immutability problem
Yeah, in my view only an 'intent' eg a request, or event in queue, or a line in a DB, should span that kind of time. Then you just version that intent
p10jkle··on Code that sleeps for a month: Solving durable execution's immutability problem
It's a hard problem; we are basically building a distributed database from scratch to solve it. You have to pick a position on the DB-design trade-off curve where you can store state with low latency, consistently, where the state is very ephemeral (long running handlers aside, generally it should only live for a few minutes). But more than anything, the 'trick' is storing service state, execution state (ie the journal), and communication between services all in the same distributed log - that means you can commit all 3 atomically
p10jkle··on Solving durable execution's immutability problem
In general, if you're changing database schemas, you need a code version running that supports old and new. If you have some old code running to serve in-flight requests, you probably need to wait for those requests to complete. This is one of the main areas that makes me feel that requests > a few hours are a real problem (and that we shouldn't be writing workflows this way)
p10jkle··on Solving durable execution's immutability problem
It's probably the next on the list :)
p10jkle··on Solving durable execution's immutability problem
if you replay the request on the new version then you might encounter new steps that don't match what you have in the journal. temporal users know this pain well...
p10jkle··on Solving durable execution's immutability problem
If don't keep old code around, you are forced to either drop pending requests that started on them, or try and replay them on the updated code, which can't be known to be safe. So it's better to keep it around, but this brings in a new set of infra and security challenges, eg where will it run, what will it cost, will it have a vulnerable dependency, etc
p10jkle··on Pkl, a Programming Language for Configuration
This is a full programming language! It’s just designed for exactly this goal.
p10jkle··on Pkl, a Programming Language for Configuration
I absolutely loved it and I’ve been extremely impatient to see its release. I used it to generate k8s manifests, Terraform, all infra config files. Very flexible and fun to use
p10jkle··on Pkl, a Programming Language for Configuration
Yes, I was using it when I joined Apple in mid 2020
p10jkle··on Pkl, a Programming Language for Configuration
I'm not sure I'd agree its solved already, yes we have some languages (Dhall or Cue probably lead the pack) but I hugely prefer Pcl's syntax and approach. It's like saying that C solved systems languages, so there's no reason to have Go or Rust
p10jkle··on Pkl, a Programming Language for Configuration
For me, that's where all the power of the language comes from. It's like writing your config in Go or Python (which I think is also a great approach) except its designed from the ground up for this use case of config generation.
p10jkle··on Pkl, a Programming Language for Configuration
If you're interested, there's now an unofficial community Discord: https://discord.gg/vDhhCT24
p10jkle··on Pkl, a Programming Language for Configuration
I used this extensively when I worked at Apple and LOVED it. So excited that its finally out
p10jkle··on A Partisan Solution to Partisan Gerrymandering: The Define–Combine Procedure
It's also an incredibly effective way of getting fascists elected - who can then use their platform to gain legitimacy and support. I look at Europe and see fascist parties gaining seats in every country - except the UK, because in the UK FPTP will ensure they never get one. Even UKIP, never won a seat, despite being hugely influential populists.
p10jkle··on We got 400 LoC StepFunctions to 40 LoC TypeScript by making Lambdas suspendable
yep no busy waiting for calls and sleeps within the system. However, 'sideEffects' where you basically just commit the result of an external operation still block the Lambda. But we're thinking about exposing a `fetch` api directly from the runtime (ie, we do a call on your behalf) that could in theory sort that out
p10jkle··on Rethinking serverless with FLAME
I'm working on something that I think might solve the problem in any language (currently have an sdk for typescript, and java in the works). You can avoid splitting an application into 100s of small short-running chunks if you can write normal service-orientated code, where lambdas can call each other. But this isn't possible without paying for all that time waiting around. If the Lambdas can pause execution while they are blocked on IO, it solves the problem. So I think durable execution might be the answer!

I've been working on a blog post to show this off for the last couple of weeks:

https://restate.dev/blog/suspendable-functions-make-lambda-t...

p10jkle··on Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python
Nice! If I were to write a test for invariant aspects of the function (eg, it produces valid json), will the system guarantee that those invariants are fulfilled? I suppose naively you could just do this by calling over and over and 'telling off' the model if it didn't get it right
p10jkle··on OpenTerraform – an MPL fork of Terraform after HashiCorp's license change
https://www.hashicorp.com/trademark-policy - surely this is a violation?
p10jkle··on 32“ E Ink screen that displays daily newspapers on your wall (2021)
No, this isn't true. I run the Visionect software suite (https://docs.visionect.com/VisionectSoftwareSuite/index.html) on a Raspberry pi and the screen talks to it over localhost. No outbound connectivity or subscription is needed.
p10jkle··on 32“ E Ink screen that displays daily newspapers on your wall (2021)
I believe so. Visionect 32"
← PreviousPage 2 of 4Next →