zknill

I am here for it too. Java's release cycle is now 6 months (matching Go's), and Java is introducing more of the features that make the 'newer' languages pleasant to use. Features like Pattern Matching for switch[1], Virtual threads [2], and String templates[3]

[1] https://openjdk.org/jeps/441 [2] https://openjdk.org/jeps/444 [3] https://openjdk.org/jeps/430

> excepting those poor, sad professionals who have to write languages that can only be written with a special, hyper-bloated text editor [1].

> 1. I'm talking about C# and Java

I write Java in vim, and it's pretty usable. Coc.nvim and coc-java fix code completion. There are also plugins to make running tests, and the debugger usable too. I can't stand the start-up time of IntelliJ!

It reminds me of every trucking company that aren't trucking, they are "Delivering innovative supply-chain logistics solutions"

I regularly share this with teams that I work with. I think it's the clearest explanation of how to write docs that I've seen.

I see "explanation" and "how-to" conflated so regularly, as a quickstart how-to guide falls into a tangent of explaining why something works a certain way.

The promo video has a perfectly flat body of water. I'd be interested to know what impact waves have on a hydrofoil.

When you search for usage of Differential Sync you'll find lots of news reports about it being included as technology in file sync and file backup solutions.

I've not directly worked on it, but I get the impression that it's mostly used in file backup because of how Differential Sync works. In Differential Sync the server and the client both hold two copies of the data/document. One representing their own state, and one representing what they thing the other's state is. Then the differences are computed between those documents and sent to the other.

For any given client, this isn't too onerous. You'd have to hold the client and server copy of the doc. But for a server, serving many clients, you'll end up with a lot of client-specific copies of that document. For N clients, you'd have N+1 copies of the document. It doesn't take too many clients before this becomes a real problem.

So if you're backing up 1 client to 1 server, Differential Sync is fine. But if you're trying to keep tens, hundreds, or thousands of clients in sync then you end up with a lot of client "shadow" copies of the server's document.

> But this doesn't mean we should just give up on ux

It's a little unfair to describe locking as "[giving] up on UX", especially given some well known collaboration products use it quite successfully; Google Sheets cell locking, Miro element/Text locking, etc.

Ofc, it's going to depend on the scope of what's being locked. These two examples are quite finely scoped element locks.

> So use the CRDT library when building your indices?

Yeah, sure, you can build a secondary index over the data. But you're still having to decode the blob and index it. There's no version where you can index the data without using the library to expose the underlying model state (like you could if you weren't using a CRDT).

On locking, yes, it's hard. But it's not the same kind of locking that you'd expect in other parts of a system. You're locking the UI, not the actual data, so it's a tiny bit more forgiving. In general the locks aren't trying to force consistency, instead they are trying to prompt the humans to reduce the chance of conflict happening in the first place. Ofc, you still have to care about the locking, unlocking, disconnection problems, etc.

Here's a decent example/demo you can play around with in multiple windows:

https://examples.ably.dev/component-locking?space=W0V-t5oY6A...

https://ably.com/spaces

How do you handle getting the changes that one client makes onto the other clients? Are you pushing it from the server to the clients with websockets, or waiting for the clients to ask for new info, or waiting for the conflict to happen when someone else tries to make a change, or something else?

I'm thinking a lot about keeping server and client data in sync while working on our hopefully-soon-to-be-released LiveSync product[1]

[1] https://ably.com/livesync

So Last-Write-Wins (LWW) basically _is_ a CRDT, but not in the sense that anyone really expects, because they aren't that useful or intention preserving. Especially if the two writes happen in very quick succession / concurrently.

LWW becomes useful if you can:

a) help humans to see who is doing what on a doc

b) reduce the size of the change that is LWW

As you've said:

> However our LWW texts are individually small - one block/paragraph in size

This is really important, because by reducing the size of the individual element that any single user is updating you can also reduce the chance of conflicts. With a small amount of contextual feedback (like the notion icon next to the block) a lot of the problem cases are just avoided.

Clearly locking and updating the entire document would be terrible, but if you can do it on a small enough scope that others can change other elements, it can work really well.

(If you've worked on the notion things you're describing then I'm sure you know this better than I do, but just spelling it out really clearly.)

You're absolutely right, locking is a hard problem. Especially when you get into the edge cases of clients not disconnecting cleanly.

If you've ever used one of those awful early-2000s Microsoft word collaboration systems where you have to check-out the doc, and remember to check it back in, and no one can use it until you've checked it back in... it's awful!

I'm not directly in this team, but one of the teams at my company have been working on this problem. They call it "Spaces", and one of the features solves this component locking problem.

https://ably.com/examples/component-locking

I think time is a terrible proxy for effort or substance or scope.

Take the example from the article:

> Please look into this for me. Do not, DO NOT, spend more than 20 minutes on this. Please come back with whatever you have after 20 minutes.

Each person looking at this is going to achieve a slightly different amount in 20 mins. Even if we ignore that, there's no guarantee that the manager making the request is going to get what they need from 20 minutes of your effort.

In 'agile' working environments, I regularly see discovery/research tasks labelled as a 'spike' and given a 'time-box'. I often question: "What happens if we don't discover what we want to in that amount of time?", and there's generally no good answer. What actually tends to happen is the 'time-box' is expanded until we found out what we needed to. So time is a terrible proxy for effort, substance, or scope. The same is true here, the manager has some idea of what they want to find out, and you better hope you can do that amount of research in their sized time-box.

I think the later examples on this article are better:

> Are you expecting something super thorough, [...] or a quick write up?

This is much more valuable because we're talking in terms of the scope of work, not how long it will take. When anyone measures in time, there's no guarantee that you'll get what you want in that amount of time, and then what? Well, we add more time!

I work for Ably, a realtime messaging platform. I'm also the author of SQLEdge (a replication tool mentioned in the article). One of our teams at Ably just launched "spaces"[1] to make it easy to build collaborative experiences. While it's not strictly in the local-first camp, I think it's in a similar 'space' (pun intended).

I think the collaboration experiences that you see in Figma, Miro, Notion, etc should _just be possible_ for all apps. And for a lot of product development teams they probably look at what would be involved to try and build a realtime collaboration experience and think it's just gonna be either too hard, or too much work. But the 'art of the possible' as definitely moved on (as evidenced by the list of companies in the blog post that are addressing these challenges).

As a bunch of the other commenters point out, the hard-part isn't just building the code for the UI that will show collaboration experiences, the hard-part is often handling the stateful connections (websockets) and making sure the messages are delivered to the right clients. You either build a stateless/polling mechanism over HTTP, or you have to deal with trying to scale and load-balance websocket connections. Neither approach is particularly easy.

[1] https://ably.com/spaces

log/slog package essentially delegates writing log messages to some "handler" interface. The key method is:

    Handle(context.Context, Record) error

This method has access to the context, which means you can get the logging handler to extract values from the context. Instead of storing the logger on the context, you can extract the traceId, etc values from the context and log those.

It's a little bit involved to write a whole logger from scratch, but you can 'wrap' the existing logger handlers and include values from the context relatively easily.

There are examples in this project (which aims to help solve your usecase): https://github.com/zknill/slogmw

Assuming you are logging from some package that's shared over a CLI app and some webservice app; log/slog expects you to setup the slog logger with a specific handler. This handler controls _how_ events are written out, the format of them etc.

If you want to use slog, I can imagine setting up the logger with a handler specific for the CLI output in the CLI tool, and a json or text structured handler in the webservice app.

The quesiton is, do you actually want structured logging in the CLI app? Yes you probably want to print something out, but is it _structured_ in the sense that slog expects? Or is it just some output.

If it's not really structured, then you probably want some other interface/library that better represents the logging you want to do. Slog will push you towards structured key-value pairs, and you might find yourself fighting against this in the CLI app.

The new structured logging library is a great addition, its nice to have structured logging in the standard lib.

It's easy to get started with log/slog and one of the built in handlers, but as soon as you want to change something the library design pushes you towards implementing an entire handler.

For example, if I want the built in JSON format, but with a different formatting of the Time field, that's not easy to do. It's not obvious how to change the built in handler.

I wrote slogmw[1] to solve this problem. It's a set of middleware and examples that make it easy to make small changes to the built in handlers without having to write a whole new handler from scratch.

[1] https://github.com/zknill/slogmw

The new structured logging library is a great addition, its nice to have structured logging in the standard lib.

It's easy to get started with log/slog and one of the built in handlers, but as soon as you want to change something the library design pushes you towards implementing an entire handler.

For example, if I was the built in JSON format, but with a different formatting of the Time field, that's not easy to do. It's not obvious how to change the built in handler.

I wrote slogmw[1] to solve this problem. It's a set of middleware and examples that make it easy to make small changes to the built in handlers without having to write a whole new handler from scratch.

[1] https://github.com/zknill/slogmw

Seems like there's 3 groups of opinions on ORMs:

Firstly (1); "I want to use the ORM for everything (table definitions, indexes, and queries)"

Then second (2), on the other extreme: "I don't want an ORM, I want to do everything myself, all the SQL and reading the data into objects".

Then thirdly (3) the middle ground: "I want the ORM to do the boring reading/writing data between the database and the code's objects".

The problem with ORMs is that they are often designed to do number 1, and are used to do number 3. This means there's often 'magic' in the ORM, when really all someone wanted to do was generate the code to read/write data from the database. In my experience this pushes engineers to adopt number 2.

I'm a big fan of projects like sqlc[1] which will take SQL that you write, and generate the code for reading/writing that data/objects into and out of the database. It gives you number 3 without any of the magic from number 1.

[1] https://sqlc.dev/

It's true for protocol version 1 that only the committed data is sent on the replication connection.

In protocol version 2 (introduced in postgres 14) large in-progress transactions can appear in the new "Stream" messages sent on the replication connection: StreamStart, StreamStop, StreamAbort, StreamCommit, etc. In the case of large in-progress transactions uncommitted data might end up in the replication connection after a StreamStart message. But you would also receive a StreamCommit or StreamAbort message to tell you what happened to that transaction.

I've not worked out what qualifies as a "large" transaction though. But it is _possible_ to get uncommitted data in the replication connection, although unlikely.

https://www.postgresql.org/docs/15/protocol-logicalrep-messa...

The logical replication protocol sends a series of messages that essentially follow the flow that a database transaction would.

i.e. a stream of messages like: "BEGIN", "[the data]", ["COMMIT" or "ROLLBACK"].

So any application that listens to the Postgres replication protocol can handle the transaction in the same way that Postgres does. Concretely you might choose to open a SQLite transaction on BEGIN, apply the statements, and then COMMIT or ROLLBACK based on the next messages received on the stream replication protocol.

The data sent on the replication protocol includes the state of the row after the write query has completed. This means you don't need to worry about getting out of sync on queries like "UPDATE field = field + 1" because you have access to the exact resulting value as stored by Postgres.

TL;DR - you can follow the same begin/change/commit flow that the original transaction did on the upstream Postgres server, and you have access to the exact underlying data after the write was committed.

It's also true (as other commenters have pointed out) that for not-huge transactions (i.e. not streaming transactions, new feature in Postgres 15) the BEGIN message will only be sent if the transaction was committed. It's pretty unlikely that you will ever process a ROLLBACK message from the protocol (although possible).

The writes via SQLedge are sync, that is we wait for the write to be processed on the upstream Postgres server. So it operates as if SQLedge wasn't in the request path from application to Postgres. The writes to Postgres are only reflected in SQLite when the data is received back from the Postgres server on the replication slot.

This means writes are eventually consistent, currently, but I intend to include a feature that allows waiting for that write to be reflected back in SQLite which would satisfy the 'read your own writes' property.

SQLedge will never be in a situation where the SQLite database thinks it has a write, but that write is yet to be applied to the upstream Postgres server.

Basically, the Postgres server 'owns' the writes, and can handle them just like it would if SQLedge didn't exist.

Yes, absolutely, and this is going to be one of the hardest tech challenges to solve. I've thought a little about it, and it's probably unrealistic to think that we can translate every single PG statement into a SQLite one, especially when PG has extensions. So we're probably destined to use the local SQLite database for queries we can parse and understand, and forwarding all the others (both reads and writes) to the upstream PG server.

This slightly breaks model of having a local copy serve data faster, but if only the minority of queries use a format that we don't understand in SQLite then only that minority of queries will suffer from the full latency to the main PG server.

If you're talking about the case where a local database forwards a read to the 'leader', and that read is put on the leader's log, but that read is only served when the read is replicated in the log back to the original local database then: It's actually less about the the leader caring about the read, and more about creating a point in time that the read will happen.

The database is using the read in the log as the point-in-time where that read happened. The writes before that point will be viewed by the read, and the writes after won't be.

By sending the read to the leader, when we receive that read back, we know for sure that all the writes that should have been replicated to our local copy have been replicated (because our read has made the full round-trip that all our writes would have, so we're guaranteed to have received all the writes before that read's "point-in-time").

This design is really similar to the approach taken by litefs[1] and turso[2]. Writes are forwarded to the primary (like you would with a write-through cache) and a local copy of the data is held in each region. These local copies are only guaranteed to be up-to-date with the most recent write forwarded by that local copy.

[1] https://fly.io/docs/litefs/proxy/ [2] https://docs.turso.tech/reference/data-consistency#on-the-re...

The optimistic concurrency is a feature of SQLite, not their sqld server.

And I think there are other approaches to replication that still retain the embedded aspect; litefs, litestream.

I guess I'm thinking, if it looks like postgres, and it speaks postgres (wire protocol) and it has features or replication like postgres, why wouldn't you use postgres?

I thought one of the main attractions of SQLite was that it was embedded, so specifically didn't need an HTTP round trip to query the DB.

It seems like this project removes that benefit, so what is it solving for or adding instead? What's the upside?

I built https://packagemap.co for working with Java code. It builds maps of your packages.

If you try it out, I'd love to get your feedback.

I built https://packagemap.co which is similar and designed for java. I'd love to get your feedback on it.

Hey, cool idea.

I built something similar for working with Java code, but it could be extended to other languages.

https://packagemap.co

I've seen engineers find it hard to work out if their code is organised well together. So I built this tool to show them visually.

There's a particular writing style that ChatGPT tends to produce that commonly starts sentences with "As a [xyz]..." and "By doing [abc], we achieved [xyz]..." , it also tends to hedge sentences by adding in words like "potentially" and "maybe".

I wonder if the OP used ChatGPT to help them write this post.