Better Gaussian Splatting in Julia | Hacker News Reader

Better Gaussian Splatting in Julia

pxl-th.github.io

126 pointspxl-th27 comments

I'm still very fascinated by gaussian splatting, but the issue I always run into is the need to use colmap to generate the camera positions and get the initial point cloud. Which colmap effectively still needs Nvidia for dense point clouds last I checked. I'm curious though does anyone have anything else they've tried that worked well? I've tried a few different things (most of the time getting stuck on just compiling the thing) so I'm curious if anyone has any suggestions that I should focus more heavily on getting working. I have some poor quality videos I'd like to try and use and I'm willing to put in some manual work, but it feels like it's all automatic or nothing. Thanks for any suggestions that any of you have.

I know there have been papers and research projects about GS without COLMAP and GS from videos. But, I don't have links handy.

Best I can do is link you to https://x.com/RadianceFields and https://radiancefields.com/ They have all the news about GS every day.

Thanks, I've been exploring a lot of the papers and seeing what I can get working. Some of it really has been interesting, but I just never seem to be able to get it to work well. I might just need to throw myself at it all a bit more. Thanks for the links though, I'll check them out.

TFA mentions this project has backends for :

AMD GPU (AMDGPU.jl) NVIDIA GPU (CUDA.jl) MacBook GPU(Metal.jl)

I'm sorry, but the article doesn't cover anything about the preprocessing steps that are required for 90% of gaussian splat implementations. That preprocessing step is almost always colmap which takes pictures or videos and uses SfM to figure out the camera parameters and their position in the 3d scene, it then does basic point reconstruction and dense point reconstruction. That info is what gaussian splats take in, it converts the points from a colmap project into gaussians and then optimizes them through differentiable rendering using backprop to optimize the gaussians to better match the images at the calculated positions of the camera from colmap. Looking over the project files also shows that they consume colmap projects, my question wasn't about the gaussian splat implementation, it was about the defacto standard colmap preprocessing step that is also required.

What hardware do you have? I've been using colmap on my MacBook. It's been working alright.

It's been a while since I've tried to run it, but it's a mix of Nvidia and AMD GPUs and only AMD CPUs (everything portable is all AMD though and that's where my gripes are.) I'm sure the opencl backend has gotten better over time, but it always seems to drag behind the cuda backend. At this point I wish we could just have a vulkan backend and be done with it.

Thanks for bringing that up though, after a quick search that I guess I haven't done in a long time, it appears that colmap does now have a hip backend. So I guess that's going to be my weekend now.

Having this work on Apple metal is huge for the accessibility around training splats. I remember two years ago I needed an RTX Nvidia GPU to run things like splatfacto on nerfstudio. Very neat optimizations as well like the depth map and sky dome, 3dgs has come a long way. Awesome!

Oh, a fellow Ukrainian! Was pleasantly surprised to see that fountain on HN — I was just walking around it a few days ago, very nice and peaceful spot this time of year.

Дякую! :)

I thought the picture on your home page was very cool, and then I moused over it. Absolutely incredible.

Fantastic demos in Julia, what a treat.

Hey Anton, great work as always :)

Hi Christian! Thank you! :)

Great work

Better? A turn on the spot shows a wreck everywhere but forward.

do you know how gaussian splatting works?

Those are the areas that are occluded in the input images. If you look at the bushes, you can see that the front is clean while the back is a mess. Probably the starting position could be chosen a bit better.

The path to left and right is occluded? By what?

By not taking photos of it... what else? The data wasn't there, so it couldn't be generated

Actually, this might be a lack of understanding, look up what gaussian splatting is.

> By not taking photos of it...

That's not occlusion.

lol, there can be more than one explanation, you're being purposely obtuse.

Occlusion can block you from taking a photo.

Not taking a photo can block you from taking a photo.

Sometimes, there are two things that happen!

Seriously, you'd understand if you actually bothered to look up what Gaussian Splatting is

This raises the question, do any of these projects have workflows for combining images taken from different positions? Or does that just work out of the box? (I have no machines atm with anything better than 10yo eGPUs ;~<)

Yes the whole field could more or less be described as combining images taken from different positions :)

As I understand the typical workflow starts with a stack of frames which have a small offset in position /orientation which the model is able to derive due to sufficient similarities to the adjacent frames; if I created three stacks like that which are internally self-similar but were taken from three viewpoints 20 meters apart, so perhaps only a small portion of the scene is common between them, could the model still orient them into a single scene? I suppose you could just capture the "transition" frames as you move your camera from one viewpoint to the next as well and then it would be treated as a single path?

The article includes a visualization of the different positions that images were taken from: https://pxl-th.github.io/video/camera-frustum.mp4 It's just that they're all bunched around the statue and the starting position for the rendering is a bit farther away, where there's direct line of sight to spots not covered by those images.

this is amazing!

Reply on news.ycombinator.com