That's all to say, have you considered that framing, and if so, have any opinions on why more general observability tools haven't gone that direction?
43 karma · joined June 18, 2024
That's all to say, have you considered that framing, and if so, have any opinions on why more general observability tools haven't gone that direction?
We don't really use images for acceptance testing, it's more functionality. I don't really care where the button is, just that if I click it, it does what it says it does. To that end, we already have multi-tenancy in the application (I highly recommend every app build this in, if only for testing; then you don't need to do the Postgres template1 seeding hacks), so we spin up a tenant for a test, do all the setup with the API (all the Given statements), playwright to do the validation of a feature (the When statements), and finally validation is a mix of Playwright and API (the Then statements).
Sorry if that isn't super clear; maybe I could put together a little demo repo or something if people have any interest; I'm hoping to do that after I introduce typespec and dbml (basically spec/{acceptance,api,db}).
I am also really interested in standardizing the REST interface with typespec, and the database schema with DBML. I think creating these closed-loops with high-level specs is one of the best ways to make an AI project ownable by a team.
I've been playing around with using a hidden markov model informed by a UI state event stream, with the end state fed into the Agent as a hint on each message turn. Then the Agent can make a tool call to add events to the HMM. This has been really interesting, but I haven't struck the right balance to make it actually feel good for the user yet
1. https://www.reddit.com/r/myog/comments/1k3stln/ultralight_13...
On the other hand, I'm playing with the idea of a platform which provides a Gaussian Splat based Digital Twin of an environment so other systems can utilize it to share location-based information. Even though I don't think it'll be possible to build without utilizing Gaussian Splatting; splatting may not end up in any of the pitches or advertising directly.
Gaussian Splats are able to handle more heterogeneous information sources, allowing more sources to help splat an environment. Devices like drones, surveillance cameras, or autonomous systems can be used to create or incrementally update a Gaussian Splat; and there's interesting work to allow them to locate themselves within the splat, not just to show themselves but also to place vision ML outputs into it (such as object detection or segmentation results).
Up till now nearly all digital representations of physical environments are either based off the original designs (by things like CAD or BIM files), or are an approximation of the environment (from photogrammetry or Lidar scans). CAD and BIM files suffer from drift, the real environment almost never perfectly matches the design files, small (and large) changes are made; and many times those files aren't even available if the structure isn't new. Photogrammetry and Lidar scans struggle because their output is a pointcloud, and it's very difficult to accurately mesh a pointcloud (Matterport only partially solved this problem and sold for $1.6B). Gaussian Splats overcome these issues; they're comparatively easy to generate for any environment, and allow for very accurate and easy viewing from any angle.
I think the Digital Twin space will be turned upside down, and they could potentially even cause huge changes in autonomous and semi-autonomous factories, warehouses, and depots. A single Gaussian Splat could be the source of truth that many autonomous vehicles update through their separate SLAM systems. Operators then would have access to this splat (and it's history) as a source of truth for the environment. Then, using techniques like iComMa[1], it may be possible to directly align XR devices into the Gaussian Splat; allowing operators direct access to location-based information generated by the environment.
That's a lot of words to say: Gaussian Splatting is a very neat new technology that could really underpin many future technologies, I'm really excited about it