33 karma · joined November 10, 2023
Agreed codegen works fine here too by the way, but it feels kind of clunky to me (it’s something I don’t love about Go, although I know it’s also an important part of the ecosystem and is popular).
@hatchet.task()
async def my_task(...) -> SomeOutputType:
return SomeOutputType(...)
## imagine this is an API handler:
@api_handler("/some/path")
async def handle() -> ...:
result = await my_task.run(...)
# we now know `result` is of type `SomeOutputType` without any sort of type assertion, etc.
Admittedly, I'm not a Go expert, nor am I a programming languages expert. But I do feel that this type of behavior is really only possible (with nice ergonomics) with generics, and it's always been upsetting to me that somehow Python's type system feels more complete than Go's in this arena, or at least it has until more recently.Maybe this falls into the 1% of cases, but I'd suspect this sort of thing is more common than that.
Edit: I should have mentioned - in the Python example above, `@hatchet.task` is generic with the output type of the task it wraps.
Also, to re-emphasize: we do this rarely, but it's been helpful the times we've done it
I, at least, don't know of a perfect fix here. Re: the original comment - Postgres will also error on deadlocks after it detects them without setting your isolation level to Serializable, but I agree with you that often retrying doesn't help, and could even cause cascading / snowballing failures if you have a backlog of retries piling up because of deadlocks.
I don't know if there's a good solution, really. We've fixed deadlocks incrementally over time as we've found them, which has worked pretty well, but of course that means also needing to deal with the "finding" part, which has generally come in the form of lots of `deadlock detected` log lines and errors (and retries accompanying those).
One thing that might be worth auditing is why there are two different bits of application code that are updating the same rows in two different tables in different orders. I know it's a contrived example, but it seems like it could be a code smell to me. Maybe this is the kind of thing that arises when two different subteams are working on the same database and are largely siloed.
Alexander will likely have more thoughts here as well, just my two cents!
One small addendum here is we've had a lot of success performing joins in memory in a few very specific situations where the alternative is a single, often overcomplicated query. I've heard / seen advice many times in the past about performing fewer round trips to the database being something to optimize for (often good advice!). Sometimes this is taken too far, resulting in overly-complex queries requiring complicated JOIN or UNION logic, CASE logic, and so on.
We have a couple of places in our codebase where we perform two or more simpler queries independently instead, and then loop through their results and use maps to match the relevant rows. Conventional wisdom often suggests this path will hurt performance because of the extra database round trip in addition to the loops needed to perform the join, but it is actually beneficial in these cases because of more predictable query planning behavior. We use this trick sparingly, but it can be helpful in a pinch.
Note that some ORMs will also do this for you in the background, which we don't necessarily endorse, and we try to use this sparingly when writing a single query on its own is not realistic.
Happy to answer more questions beyond this!
Mostly to learn some Rust and because I thought most of the features of Splitwise worth paying for would be fun to build. Been loving working in Axum and getting to implement some fun database things