diarrhea

This is extremely alarmist. If you are on Mac, developing anything at all with Python, those are going to be your tools. If you randomly guessed at developer setups, that is going to be one of the most common setups as well.

What's next, I use a computer so someone could just upload a virus somewhere in hopes I happen upon it, which would be dangerous?

I have dotfiles which not only specify Python, but also its version and some packages. That protects against typosquatting! I will never type those package names out manually again. And this is all in Nix (btw), so all hash-pinned!

I've not used Redis Streams, but it might work. I've seen folks advise against PG, in favor of Redis for job queues.

> using a table instead of LISTEN/NOTIFY

What do you mean? The job queue is backed by a PG table. You could optionally layer LISTEN/NOTIFY on top.

I've had success with a table with compound, even natural primary keys, yes. Think "(topic, user_id)". The idea is to allow for PARTITION BY should the physical tables become prohibitively large. The downsides of PARTITION BY don't apply for this use case, the upsides do (in theory - I've not actually executed on this bit!).

Per "topic", there's a set of workers which can run under different settings (e.g. number of workers to allow horizontal scaling - under k8s, this can be automatic via HorizontalPodAutoscaler and dispatching on queue depth!).

Your version makes sense. I understood the OP's approach as being different.

Two (very, if indexed properly) short transactions at start and end are a good solution. One caveat is that the worker can die after t1, but before t2 - hence jobs need a timeout concept and should be idempotent for safe retrying.

This gets you "at least once" processing.

> this obviously has the drawback of knowing how long to sleep for; and tasks not getting "instantly" picked up, but eh, tradeoffs.

Right. I've had success with exponential backoff sleep. In a busy system, means sleeps remain either 0 or very short.

Another solution is Postgres LISTEN/NOTIFY: workers listen for events and PG wakes them up. On the happy path, this gets instant job pickup. This should be allowed to fail open and understood as a happy path optimization.

As delivery can fail, this gets you "at most once" processing (which is why this approach by itself it not enough to drive a persistent job queue).

A caveat with LISTEN/NOTIFY is that it doesn't scale due to locking [1].

[1]: https://www.recall.ai/blog/postgres-listen-notify-does-not-s...

Yes, and that cannot work: if a task is unlocked but in 'executing' state, how was it unlocked but its state not updated?

If a worker/processor dies abruptly, it will neither unlock nor set the state appropriately. It won't have the opportunity. Conceptually, this failure mode can always occur (think, power loss).

If such a disruption happened, yet you later find tasks unlocked, they must have been unlocked by another system. Perhaps Postgres itself, with a killer daemon to kill long-running transactions/locks. At which point we are back to square one: the job scheduling should be robust against this in the first place.

This introduces long-running transactions, which at least in Postgres should be avoided.

> What’s the rationale for making it all free?

The writing is on the wall for Leetcode-style interviewing. The signal-to-noise ratio is diminishing in the age of AI (cheating). These sorts of puzzle challenges might no longer play a meaningful role going forward.

IIRC it is just having locks with exclusive constructors, which take previous locks’ guards (by ownership?).

That way you can never lock lock B if you have not received a guard aka lock from lock A prior. Ensured on the type level.

I suppose doing this at scale is a real challenge.

Thank you! Very insightful, especially the forum link and the observation around UUIDs bloating indexes.

This is very exciting stuff.

I’m curious: When you say FOR UPDATE SKIP LOCKED does not scale to 25k queries/s, did you observe a threshold at which it became untenable for you?

I’m also curious about the two points of:

- buffered reads and writes

- switching all high-volume tables to use identity columns

What do you mean by these? Were those (part of) the solution to scale FOR UPDATE SKIP LOCKED up to your needs?

Yes, but that is not garbage.

> It just litters perfectly reasonable python code with async/await

Yeah. As an API consumer I would not expect a PDF API do IO, hence be async. Have the library be sans-io, the interfaces sync and callers from async code handle IO on their end, offloading to IO threads.

Async is also referred to as “best practice”, but it’s just a tool, for specific use cases. And I say that as an “async fan”!

That said, perhaps it’s easier nowadays to just do async by default, as you say. The real world is async anyway, so why not program closer to that reality.

I’m curious about the async aspect of this. I was under the impression PDF processing like OCR is purely CPU bound. OS file I/O interfaces are sync, so async does not help. With GIL, so single threaded Python, I can’t see how async improves performance for the PDF use case. Only parallelism helps, and concurrency doesn’t. When would it yield back to the event loop when it’s busy number crunching?

Interesting, thank you for the example. I'm mostly used to how Rust handles this, and in its approach individual items such as functions need to be "standalone sane".

    func foo[T any, V any](x T) V {
        return x.y
    }

would also not fly there, because T and V are not usefully constrained to anything. Go is the same then. I prefer that model, as it makes local reasoning that much more robust. The C++ approach is surprising to me, never would have thought that's possible. It seems very magic.

> Sadly, Go's generics are a bit of a basket case in terms of usefulness since they don't support compile time duck typing like C++ generics,

What are you referring to here? Code like

    func PrintAnything[T Stringer](item T) {
     fmt.Println(item.String())
    }

looks like type-safe duck typing to me.

Your first paragraph is a great description of what I am currently feeling as well. There just isn’t anything there. I’m familiar with Python, Rust, C# and other random languages. Go just has nothing to offer meanwhile. I am struggling with nil and zero values every day, the lack of enums, and the bizarre error handling (wrapping etc.).

I’m waiting for it to click but nothing yet. Channels in combination with effortless (uncolored) asynchrony are maybe it. I haven’t reached that point yet though as I’m still struggling with just expressing my domain, without having invalid and primitive values, constructed and passed around by design, all over. I suppose it is not possible. Every time I see “url string” in a function or struct I die a little inside.

Yes. The concept of references (which includes that of null pointers) and that of optionality (which includes that of nothingness) are orthogonal. The former is just an artefact of computer architecture, the latter is everyday business logic. Go mixes the two into a single concept, which is painful.

As a newcomer to Go, I find myself struggling to express business logic all the time. Zero-valued types make the situation even worse. You’re constantly constructing invalid values of types (nil/zero-valued), by design. Go is “so easy”, but when you inevitably run into one of the footguns it’s “yeah just don’t do that”.

One of the primary reasons for people to dislike Python is its dynamic typing. When Go came out, that was totally fair. But since then, Python has evolved and improved massively. It/mypy now supports type-safe structural pattern matching, for example. It’s very expressive, and safely so.

Meanwhile Go barely evolved. Generics landed just recently. They’re only now experimenting with iteration, lifting it from a purely magic, compiler intrinsic concept. And still, no enums of course. The “type system” are structs, or very leaky type wrappers (nowhere near the safety of Rust new types, for example). People are obsessed with primitives.

I can see the appeal of a simple, stable platform, but Go really ran too far with that idea.

Do you have a concrete example by any chance?

Yeah, the landscape is horrible.

I created NS records on example.com to delegate all of home.example.com to a wholly different DNS provider. That provider then manages (all of) that zone, but nothing more (important records such as MX remain on example.com).

Lua can also be massively faster than Python (LuaJIT).

> A trait is object-safe as long as no trait method returns the implementing type (like T::clone() -> T)

That, and much more: https://doc.rust-lang.org/reference/items/traits.html#object... .

> having the intuition of how fast computers can work

I wish I never developed it. It turned what used to be mild annoyances into questioning my career choices.

A coworker of mine dislikes it as it bundles everything into a single binary. For example, to have ACME DNS-01 challenges for certificate issuance working, I need to compile in a Google DNS-specific plugin.

But then it... just works. Try the same with most other web servers/proxies and you're in for a world of pain. Having this much functionality bundled into a single binary is as much a curse as it is a blessing.

That said, having your own little 'Cloudflare Workers' in the form of Nginx Unit with wasm sounds great. Not sure Caddy can do that.

From their perspective, what are you going to do?

Say you implemented a new function and missed some clear edge cases in the new tests. Or perhaps there's no tests written at all, because it was half-assed.

You will have to speak up in a peer review. I'd speak up just to have it written down. Perhaps you just forgot to `git add` that part, I don't know. It'd be professional negligence not to at least bring it up! If your reply is then, 'I'm aware, and didn't do this for reason X', that's perfectly fine, if reason X is at least slightly better than couldn't be arsed.

Companies generally follow decision-driven data principles.

Azure Cloud isn’t mentioned in the article.

Empty/unset environment variables not, by default, being cause for immediate error and abort is the billion dollar mistake of the DevOps/infra world.

I feel like a filesystem with native snapshot support (zfs, btrfs), replicated once (also native in zfs) obsoletes conventional 3-2-1 backup systems. It’s technically only 2 versions, but you’re protected from all the same failure modes.

Hilarious coincidence. I had and loved that exact same model for the same reasons. Until one of the dials broke and I discovered how utterly irreparable the thing is. Had to get rid of it and indeed, it’s impossible to find similarly simple models. Oh well!

I wish more people used asserts like that. Such a wonderful tool for documentation. It complements the type system and testing really well.

I’ve been using catchall email for everything for well over 5 years now, and it hasn’t been useful once since. Regular spam filters from my provider and occasionally hitting “Unsubscribe” once seem to do the trick.