splatgallery.com
It's a super early stage and new models are coming it fairly often.
Would love to get feedback from the HN community on how to improve it!
It's hard to know what Photosynth could have become. It was shut down by Microsoft for unknown reasons. If it was open source it might have evolved and might still be around.
Gaussian Splatting has multiple implementations on both the training/generation side and the rendering side.
I think SplatGallery is an exploration of how people use the technology and what can it be useful for today and what is the next step.
Hard to believe, but the main technique is just over a year old (built on the shoulder of giants). This is the seminal paper for it [1] and here's a three-hour video so you really understand it [2] (and great how-to-really-read-a-paper-if-you-are-serious video). So what's great is a bunch of labs saw that and started building on it, so in the last three months there have been so many great improvements. And so much work done openly!
This has a great video roll [3] of some recent work, including use in construction and forestry.
If you have an Apple Vision, try MetalSplatter [4] and you will get an idea of how OMFG this stuff is.
We are in such a rich time of new compression knowledge and reality into weird representations!
I've been trying to evangelize it, but it takes so much foundation to understand how interesting it is. Many people (even software engineers and computer scientists) don't understand traditional 3D rendering pipelines and meshes/triangles and lighting. So you have to explain that, then concepts of spherical harmonics, gaussians with affine transforms, and the miracle that happens when you sample millions of them using raycasting at 800 fps. The neural network approaches go at 2fps.
In ideating and communicating the possibilities, I try to focus on the workflows...
Capture: we can sample lighting and depth of scenes with our phones or fancy cameras or drones; we can also use photogrammetry (algos/math to create depth fields from photos). This isn't specific to 3DGS, but 3DGS empowers it to be useful. So we have this tech where we can more easily capture objects and environments, edit them, and play it back. 3D captures has been around a long time (e.g. I worked on the Immersion Microscribe in the mid0-90s [4] and this is what machine vision used to be about), but we didn't have techniques to infer structure from the point clouds.
Processing: magic math turns this into a bunch of Gaussians with a transformation matrix (bell curves of different shape floating around) which represent the structure. Literally the components are spherical harmonics (color), density (alpha), variance, translation, rotation. Scenes will have many hundreds of thousands to tens of millions of them.
I tend to mention how LLMs capture aspects of knowledge into a big sea of weights that get computation applied to them and that this is similar very abstractly; and researchers have worked with Neural Radiance Fields. But what's great about Gaussian+Transform is that this is you can actually get an intuition of what's going on -- and the editors let you edit with the gaussians and filter/prune them. You can't do that with an NN (have intuition and direct manipulation).
Rendering: those objects are sampled and drawn on at interactive rates. You can represent scenes or objects. You can commingle them with traditional 3D assets. Works on recents phones very well. So this tech is broadly available now for playback.
The thing about it is that they lack structure. It's can be very ghostly and ethereal. So I think in the near term, this tech will be fantastic for customization/personal object capture/integration with scenes... but not for simulation but for human communication. As noted, the forestry and construction videos hint at this. Also product displays in website -- here's a Shopify plugin [6].
I have so much to say about it, but will stop here =)
[1] https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
[2] https://www.youtube.com/watch?v=xgwvU7S0K-k&t=1s
[3] https://www.youtube.com/playlist?list=PLrhy9mGYkm0aZnjL-4OpO...
[4] https://apps.apple.com/us/app/metalsplatter/id6476895334
[5] https://revware.net/microscribe-portable-cmm/microscribe-i-p...
2. Support for any emerging formats that improve on .ply (https://github.com/nianticlabs/spz looks promising)
3. Better navigation controls (or at least document what you already support)
More formats are definitely on the roadmap. At first I'd like to add full support for higher order of spherical harmonics and then I wanted to take a look at other methods, too. Maybe 2DGS and/or SMERF would make sense for comparison purposes.
Explanation of controls makes sense, too. I'll add that.
Regarding WebXR, that's currently not on my roadmap. However, if the demand is there I'd consider adding it
* I discovered I can translate with ctrl-mouse, but maybe add WASD keys as well?
* I discovered "L" brings up a camera rotation display. The sliders rotates on weird axes (wrong space?), so not too helpful of a UX. Clicking the help magnifier says "Open Filter with CMD+SHIFT+L" but on Safari that opens the sidebar.
* Simple help display for either of those too might help
EDIT: * Tried it on an iPhone 13 Pro... touch controls work great and it is so smooth an good looking. In landscape mode, the header takes up a lot of space. A full-screen button would be great.
- Adding better controls (and an explanation) makes sense!
- Good job discovering the L control. That's actually something that's not supposed to be public. I use that tool to find the correct orientation of splats before publishing them. But once I find those values saved in the DB, there's no use for that tool anymore. When I make it available to the users I'll defo make the UX clearer.
- Good point about landscape mode. I'll fix that.
The two category buttons at the top should be a tags not buttons. They should change the url history so the back button works and so I can alt-click to open in a new tab.
Are there any particular niches or applications where community-sourced splats are especially well-suited? I see them mentioned on HN from time to time, but I'm curious about their primary use case at the moment.
One of my future plans is to extend it to other methods than just gaussian splatting so that we can compare methods across a wide range of captures with varying light conditions, different materials, etc..
360/180 panoramas are fun (I especially used to love the Google stereo panoramas that also contained ambient audio loop from the moment of capture) - but a splat scene is a whole different level.