maxmcd

Hard to blame the CAP theorem since this is a problem across an interface layer. If the DB knew about the data it could manage the LSM tree without issue.

Since this is about aborted writes during VACUUM then this is likely also relevant: https://www.cs.cmu.edu/~pavlo/blog/2023/04/the-part-of-postg...

Data races are covered: https://harmony.cs.cornell.edu/book/#sec-57

Threads+locking introduce data race issues into Harmony's model. As far as I can tell.

How would a distinction between concurrency and parallelism benefit the modeling of program logic?

The programming language incorporates thread+locking mechanisms.

router.blockdh100b.net resolves

so does router.blockdh100c.co

Ah cool, the note here is also interesting: https://pkg.go.dev/github.com/google/uuid#EnableRandPool

I think "no buzzwords" would be further in the direction of:

"This software let's you set up peer to peer networking between your devices, expose them to the internet, run applications on them, view useful runtime information, and integrate easily with cloud providers and infrastructure you're familiar with."

also: https://crates.io/crates/object_store

yep: https://github.com/hugomd/parrot.live/blob/f349d2788fc47ac5f...

Thanks for all your work Sean! It's been a delight to use your webrtc libs and see your impact across a broad range of technical efforts.

I think the upcoming CedarDB is HTAP? https://cedardb.com/

Clickhouse performance for Postgres workloads?

I don't think this is trivially fixed since you can still only have one writer that is holding the lease.

Your new service will come up, but it won't be able to get the write lease until the previous server shuts down. Now you have tools to detect this, stop one writer, and start the other, but the service will likely have to experience some kind of requests queueing or downtime.

Looks like the code is here: https://github.com/benbjohnson/litestream/tree/v0.5

Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193

I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.

The read replica FUSE layer sounds like a real nice thing to have.

edit: Ah, it works like this: https://github.com/benbjohnson/litestream/pull/617

> When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.

Sounds workable!

Since db cache consistency is a complex research space with things like https://readyset.io/ and https://www.usenix.org/system/files/conference/nsdi13/nsdi13... it would be great to hear more specifics about what this is giving up in order to sidestep all that complexity.

If you missed it, there was a nice story around One Million Checkboxes: https://eieio.games/blog/the-secret-inside-one-million-check...

I wonder if something similar will happen here.

@eieio please open source the Go code, would be fun to poke at.

This project has a pattern just like that to handle the inbound USPS information:

https://www.val.town/x/geoffreylitt/stevensDemo/code/importe...

I think it would be pretty easy to extend to support other types of inbound email.

Also I work for Val Town, happy to answer any questions.

Anyone want to hazard a guess about the likelihood of us ever getting high quality versions of these commercials? Are many of these likely lost to time, or is it possible that high quality copies exist somewhere?

It does not seem to speak SQL:

    $ git clone git@github.com:Sahilb315/AtomixDB.git
    Cloning into 'AtomixDB'...
    $ cd AtomixDB/
    $ go run .
    Welcome to AtomixDB
    Available Commands:
      CREATE       - Create a new table
      INSERT       - Add a record to a table
      DELETE       - Delete a record from a table
      GET          - Retrieve a record from a table
      UPDATE       - Update a record in a table
      BEGIN        - Begin new transaction
      COMMIT       - Commit transaction
      ABORT        - Rollback transaction
      EXIT         - Exit the program
    
    > create
    Enter table name: foo
    Enter column names (comma-separated): bar,baz
    Enter column types (comma-separated as numbers): 1,2
    Enter indexes (format: col1+col2,col3, ... or leave empty):
    Table 'foo' created successfully.
    > get
    Enter table name: foo
    
    Select query type:
    1. Index lookup (primary/secondary key)
    2. Range query
    3. Column filter
    Enter choice (1, 2 or 3):

It looks like records are stored in rows: https://github.com/Sahilb315/AtomixDB/blob/64c95afa8e574595c...

I do find the source to be well organized and quite readable. Especially if you runs the commands in the cli and then trace how they are each implemented.

There seem to be errors connecting to peers

Yes, good point, maybe that is the blocker.

Do you know if it's possible to get this working on macos? I believe Tailscale uses gvisor's tcp/ip lib (as their netstack lib) on macos for certain things.

Amazing, such a great use of the gvisor userspace tcp/ip stack!

Yes we do not have that. Hopefully if you're using Val Town you're getting a lot of out the interface and platform and this is just another api surface. If you want to pull your code and files and run them elsewhere that should be easy. Git sync to and from projects should be straightforward enough to implement (once we ship the public API) and our runtime is easy enough to replicate by running things with Deno directly.

This was definitely an "easier for us to run, good for the product experience" decision. Freedom comes elsewhere :)

Only "x/sys/unix" outside of tests/benchmarks

      ~/go/src/github.com/voidDB/voidDB  git:(master)
      $ go run github.com/tailscale/depaware@latest
      github.com/voidDB/voidDB dependencies: (generated by github.com/tailscale/depaware)

            github.com/voidDB/voidDB/common                              from github.com/voidDB/voidDB+
            github.com/voidDB/voidDB/cursor                              from github.com/voidDB/voidDB
            github.com/voidDB/voidDB/free                                from github.com/voidDB/voidDB
            github.com/voidDB/voidDB/node                                from github.com/voidDB/voidDB+
            github.com/voidDB/voidDB/reader                              from github.com/voidDB/voidDB
            golang.org/x/sys/unix                                        from github.com/voidDB/voidDB+
            bytes                                                        from github.com/voidDB/voidDB+
            cmp                                                          from internal/fmtsort+
            encoding/binary                                              from github.com/voidDB/voidDB/common+
            errors                                                       from bytes+
      D    fmt                                                          from golang.org/x/sys/unix
            hash                                                         from github.com/voidDB/voidDB+
            hash/fnv                                                     from github.com/voidDB/voidDB
            io                                                           from bytes+
            io/fs                                                        from internal/filepathlite+
            iter                                                         from reflect+
            math                                                         from encoding/binary+
            math/bits                                                    from golang.org/x/sys/unix+
            os                                                           from fmt+
            path                                                         from io/fs
            reflect                                                      from encoding/binary+
            slices                                                       from encoding/binary+
      LD    sort                                                         from golang.org/x/sys/unix
            strconv                                                      from fmt+
      LD    strings                                                      from golang.org/x/sys/unix
            sync                                                         from encoding/binary+
            sync/atomic                                                  from internal/bisect+
            syscall                                                      from github.com/voidDB/voidDB/cursor+
            time                                                         from github.com/voidDB/voidDB+
            unicode                                                      from bytes+
      W    unicode/utf16                                                from internal/poll+
            unicode/utf8                                                 from bytes+

Hey, I worked on this. I am the person who advocated for us not to use git and build our own thing instead. Happy to answer any questions or expound on my hopes and regrets!

This is so awesome. I've wanted something like this where you could visit all scales of the universe from tiny atoms, to the galaxy, to the broader universe. Does something like that exist in some form?

Tests are usually embarrassingly parallel. Instead of creating one test db (app_test) create many (app_test_0, app_test_1, ...). Run tests in many threads/processes, db per thread.

This works in a lot of cases. In some cases this might not address your bottleneck.

Also someone should write a real, performant, in-memory postgres storage driver. Then we can all be happy (with pg at least).

Yes, I think you are correct. In the video Boris/Baselime uses (_tenantId, _traceId, _timestamp). Will update that :)

It is a great step, but in my testing with the new JSON type if you use beyond 255 unique json locations/types (255 max_dynamic_types in their config) you will fall back to much worse performance for certain queries and aggregations. This is quite easy to hit with some of the suggestions in this blog post, especially if you are designing for multi-tenant use.

For this clickhouse wide event lib I'm working on (not worth anyones time atm) I am still using this schema https://www.val.town/v/maxm/wideLib#L34-39 (which is from a Boris Tane talk https://youtu.be/00gW8txIP5g?t=801) for good multi-tenant performance.

I hope clickhouse performance here can still be vastly improved, but I think it is a little awkward to get optimal performance with wide events today.