1,699 karma · joined February 27, 2018
Get my name on github, email is last.first @ gmail
> All you need for the backend is key-value storage with range/prefix queries;
This is true, I was able to quickly put together a Redis automerge library that supports the full API, including pub/sub of changes to subscribers for a full persistent sync server [0]. I was surprised how quickly it came together. Using some LLM assistance (I'm not a frontend specialist) I was able to quickly put together a usable web demo of synchronized documents across multiple browsers using the Webdis [1] websocket support over pub/sub channels.
[0] https://github.com/michelp/redis-automerge
[1] https://webd.is/
Based on some online anecdotal evidence, I decided to try nicotine "therapy". I bought 4mg smoking cessation mints, cut them in half with a pill cutter, and took 10-12 2mg doses per day at roughly one hour intervals. The effect was immediate and brain fog lifted in less than a week. It was like coming out of a long dream, or like I had been stoned for six months and then suddenly I was sober again. My fitness stats have exceeded where I was before I got sick.
This is just my own anecdotal experience, and there have definitely been some downsides. The mints are about $50/month. My dosage has ticked up a bit and I'm certainly addicted, at least once a day I take a full mint instead of a half for an extra kick. I'd like to taper off, but I'm not sure if I do how to know if any effects are withdrawal or resumption of the covid brain fog. I have a light caffeine habit (2 cups every morning) and I don't see the mints being any more harmful than the coffee, so I think I'm just going to stick with it.
To plug my project, I've wrapped the SuiteSparse GraphBLAS library in a postgres extension [1] that fluidly blends algebraic graph theory with the relational model, the main flow is to use sql to structure complex queries for starting points, and then use the graphblas to flow through the graph to the endpoints, then joining back to tables to get the relevant metadata. On cheap hetzner hardware (amd epyc 64 core) we've achieved 7 billion edges per second BFS over the largest graphs in the suitesparse collection (~10B edges). With our cuda support we hope to push that kind of performance into graphs with trillions of edges.
This library converts a uuidv7 into a cryptographically random but deterministic uuidv4 recoverable with a shared key. For all intents and purposes the external view is a uuidv4, the internal representation is a v7, which has better index block locality and orderability.
But still true that dense growth is not linear but quadratic to the number of nodes.
- The article says adjacency matrices are "usually dense" but that's not true at all, most graph are sparse to very sparse. In a social network with billions of people, the average out degree might be 100. The internet is another example of a very sparse graph, billions of nodes but most nodes have at most one or maybe two direct connections.
- Storing a dense matrix means it can only work with very small graphs, a graph with one million nodes would require one-million-squared memory elements, not possible.
- Most of the elements in the matrix would be "zero", but you're still storing them, and when you do matrix multiplication (one step in a BFS across the graph) you're still wasting energy moving, caching, and multiplying/adding mostly zeros. It's very inefficient.
- Minor nit, it says the diagonal is empty because nodes are already connected to themselves, this isn't correct by theory, self edges are definitely a thing. There's a reason the main diagonal is called "the identity".
- Not every graph algebra uses the numeric "zero" to mean zero, for tropical algebras (min/max) the additive identity is positive/negative infinity. Zero is a valid value in those algebras.
I don't mean to diss on the idea, it's a good way to dip a toe into the math and computer science behind algebraic graph theory, but in production or for anything but the smallest (and densest) graphs, a sparse graph algebra library like SuiteSparse would be the most appropriate.
SuiteSparse is used in MATLAB (A .* B calls SuiteSparse), FalkorDB, python-graphblas, OneSparse (postgres library) and many other libraries. The author Tim Davis from TAMU is a leading expert in this field of research.
(I'm a GraphBLAS contributor and author of OneSparse)
Using SuiteSparse and the standard GAP benchmarks, I've loaded graphs with 6 billion edges into 256GB of RAM, and can BFS that graph in under a second. [2]
https://archive.org/details/mastering-forth-by-anderson-anit...
GALAHAD: What a strange person.
https://github.com/michelp/pgfsm
Now many machines (sub-graphs of state transitions) can be defined in general, and the transition checking function checks the validity of the next state based on the table, instead of static rules in a function.
Thank you for acknowledging this. Every time Norm's work comes up on HN there is a subcurrent of comments about how his philosophy of math is wrong or dumb whose are arguments can be summed up as "Lol no infinity wtf".
Do I personally agree with his philosophy? No. But I still watched all his videos because they are entertaining, thoughtful, and his is rigorous in his definitions and examples.
This is still experimental bleeding edge stuff that's only available on the unreleased pg 18 so far (using the current dev mainline branch) but will open up a lot of possibilities in the future. Future improvements I hope to make to pg_crdt include implementing the sync() api for easy peer-to-peer synchronization and a full round-trip example application. Word is automerge 3.0 will have some nice new features and I look forward to that.
As an HN specific side note, there was a choice to make initially to use either Rust via pgrx or the automerge C API. At the time (and maybe still?) pgrx did not support expanded datum, which I think would be an amazing addition, but it was out of my scope, and my familiarity with the postgres C extension API made us choose C. Happy to discuss further improvements in this regard as I go up the Rust learning curve myself.
[1] https://www.postgresql.org/message-id/flat/647219.1736019347...
LiveJournal Orkut
Nodes: 3,997,962 3,072,441
Edges: 34,681,185 117,185,037
Triangles: 177,820,130 627,583,972
Seconds Edges/Second Seconds Edges/Second
Tri Count LL: 2.69 12,892,634 32.03 3,658,602
Tri Count LU: 1.78 19,483,812 16.38 7,156,338
Tri Centrality: 1.45 23,918,059 12.22 9,589,610
Page Rank: 7.12 4,870,953 23.14 5,064,176
Orkut was as big as I could go due to limited RAM. One of my constrains is limited access to big enough hardware to do the kinds of Graphs Of Unusual Size (billions of edges, trillions of triangles) where we can really flex the scale that CUDA support gives us. Stay tuned!