dangoodmanUT

Considering I can’t scroll down on ios safari, maybe you should…

Alright nerd-snipe snooping research post happning now!

Seems like they are using JuiceFS under the hood, with an overlay root for your CoW semantics. JuiceFS gives them instant clone (because they're not cloning the whole rootfs), while the chnages to the overlay are done as an overlayfs and probably synced back to S3 via a custom block device they have mounted into firecracker.

You can also see they are using juicefs it for the "policy" directly (which I'm assuming is the network policy functionality). iirc juicefs has support for block devices too, so maybe they are using that to back the rootfs overlay.

One concerning thing is the `/var/lib/docker` mount - i ran this in an ubuntu container, did they... attach it? Maybe that's a coincidence, but docker is not installed on the sprite by default. (the terminal is also super busted when used through an ubuntu container)

https://pastebin.com/raw/kt6q9fuA (edit: moved terminal output to pastebin because it was so ugly here)

I played with a similar stack recently, my guess is they are: 1. making some base vm, snapshotting it 2. when you create a vm, they just restore a copy and push metadata to it (probably via one of the mounts) 3. any changes that you make to the rootfs are stored on the juicefs block device (the overlay), which is relatively minimal compared to the base os. JucieFS also supports snapshotting, so that's probably how they support memory + filesystem snapshot and restore so quick

interestingly, seems they provision maybe a max disk size of 100GB for total checkpoints?

```

NAME TYPE SIZE FSTYPE MOUNTPOINTS

loop0 loop 100G /.sprite/checkpoints/active

```

fuse is definitely being used within the VMM, i can see a fuse mount and id being assigned. They're probably using juicefs directly for the policy mount because that doesn't need to be local nvme-cached, just consistent. The local-nvme -> s3 write-through runs on the hypervisor through a custom block device they attach to the firecracker vmm. This might just be the --cache-dir + --writeback cache option in juicefs. Wild guess is just 1 file per block.

guessing the "s3" here is tigris, since fly.io seems to have a relatoinship with them, and that probably keeps latency down for the filesystem

Will you have higher tier pricing plans in the future? I don't see a way to sleep them (if you mean other than idle), and the max plan has 10 running concurrently

i think firecracker, just snooping around a sprite i see a lot of virtio-mmio, which afaik CHV would be using PCI in those instances

it seems like when you snapshot, you snapshot memory AND the filesystem (immutable ftw), that's pretty awesome

i am dying to know: firecracker still? I know you have an upcoming post abt it, but i'm incredibly impatient when it comes to fool new infra

I thought fly.io snapshots weren't guaranteed to stick around? Although I can can't find the docs mentioning it, but i checked within the last few months... maybe they changed it?

waiting for someone to say "this wouldn't have happen if you chose rust"

The UI animations are so fun

yeah, nothing hits like dark sky hit

This definitely will handle large files or binary files very poorly

More related to the title, i've found the same.

I was always an aggressive pixel-pusher, so web dev took me AGES.

But with shadcn + llms I'm flying through stuff, no lie, 5-20x faster than I was before.

And i dont hate it anymore

This seems CDC based, does that mean it handles `now()` and other non-deterministic functions properly?

This is similar to what rivet (1) does, perhaps focusing more on stateless than rivet does

(1) https://www.rivet.dev/docs/actors/

A tool that I made a while back that has become a staple in everything I build, I started wanting kustomize-like functionality within environments.

The use case originally was having a simple overlay of S3 environment variables for testing so that they'd use a shared test bucket to provide agents with files, but otherwise we can use a local S3 (garage) when building.

That turned into "overlay environments": you can overlay them on top of each other just like an overlayfs, but for environment variables :)

HN link seems to strip the URL fragments, here's the direct link to the feature: https://github.com/danthegoodman1/EpicEnv/tree/main?tab=read...

dwight shrute detected

Gemini does this a lot, getting stuck generating the same tokens over and over indefinitely

Because these ai machines aren’t replacing old machines, they’re replacing old humans

I’m praying you have backups. That last paragraph gives me anxiety

The problem without having consent is that it's easy to track who is using your service. Because there's no consent, they can redirect you to login and back, and grab your identity, without you doing anything other than loading the page.

I was just sufficiently nerd sniped by this, so let me know if I’m close:

Based on what the commenter below found about sshpiper I believe that you use the ssh identity + the ip from the slot to resolve the vm target. sshpiper knows how to route the ssh identity + slot ip to the correct VM. I suspect you have a custom sshpiper plugin to do that routing.

You use the slot record indirection so you can change the ip of a slot without having to update everyone’s A records across the customer base. It also makes it easy to shuffle around vm-slot mappings within a customer. I haven’t tested, but I’m guessing this dns server is internal (coredns?), and the ips too.

I did something similar (ip + identity routing) for a project a few weeks ago. Yours is a lot more elegant with the dns indirection.

I’m no ssh expert, but in theory you should be able to ssh -J exe.dev myvm.exe.xyz for a one-liner? Or maybe you don't even need it, if that DNS server within the ssh exe.dev is the same as the public DNS. Pardon for not testing it yet!

> Zero-copy deserialization

Just a nit on this section: zero-copy deserialization is not Rust specific (see flatbuffers). rkyv as a crate for doing so in Rust is though

This script makes it easy to copy an NPM package into a `vendor/` dir.

Helps against supply-chain attacks, and also makes it easier for LLMs to investigate how packages work.

Warning: Opus 4.5 did most of the work (but we use this in prod)

> “We assumed the technology was further along than it actually was,” one executive said privately, reflecting a growing recognition that AI performance in controlled demonstrations did not translate cleanly into real-world customer environments

stop. reading. evals.

Boom, roasted.

> Semantic Kernel and AutoGen pioneered the concepts of AI agents and multi-agent orchestration

Said nobody?

I appreciated it. I have no idea why someone would downvote it other than spite

YouTube has migrated mostly to Spanner from what I hear

this would be a killer replacement for micro/circuitpython for embedded devices, assuming there's an elegant TS->MQJS transpile

Convex has been quite good so far

One thing I find about these "all in one" platforms is that they tend to lure people into a sense of "wow this is easy to use" such that they forget to check security, assuming it's covered.

This is one reason why Firebase was such a gold-mine for security researchers: everyone just forgot about security when they forgot about their backend.