olokobayusuf

Hey, I'd love to chat with you about how you power these on-device AI features (like background replacement). Function is building infrastructure for both server-side and on-device AI inference.

The goal is for devs like you to bring your original Python code, and we'll generate a library that is cached and runs on-device. See this demo: https://demos.natml.ai/@natml/blazepalm-landmark (wave your hands)

Cos to accurately match the background, you need to estimate the original camera's characteristics as closely as possible, otherwise perspective looks off.

Screen Studio!

Thank you! The audience is definitely highly technical, so this has been a very productive thread.

Would love to have a proper convo on this. With bulk pricing, I can reduce the price by quite a lot. Eventually, the goal is to have users be able to stage themselves in your property website or MLS. Please shoot me a note at yusuf@fill3d.ai !

That's fine. Path tracing in a browser is pretty impractical today anyway. Check back in a few years, when WebGPU is much more mature.

You could potentially get much faster generations (on the order of seconds) while sacrificing realism by switching away from path tracing.

True, virtual staging is a very established product in the real estate media market.

Thank you!

Seems like I'm in the middle of a controversy :D

I was hoping you wouldn't notice ;)

Don't worry, there's more options. But thank you!

Lmao I guess that's an option.

Currently it's two images free.

Potentially, by inpainting the existing object (use SD to remove it) then filling in the blank space. This makes a good roadmap feature.

I think this isn't too difficult of a problem. Technically, the objects that get added could be emissive. It could even be a switch, having an added light be on or off.

This is a very good point. Thanks for bringing this up!

Not right now, but that's a great roadmap feature. It should be trivial with today's model (object detection + inpainting). Created an issue: https://github.com/fill3d/fill/issues/2

Decorify from Wayfair is also using diffusion, same as the other folks who have built similar things in the market (InteriorAI is probably leading product here). We'll see where this goes :D

Would love to get hands on that dataset, how can I reach you? Or, shoot me a note at yusuf@fill3d.ai

Yes that's correct. And for using it, you have to draw rectangles before you can add a prompt, similar to Photoshop's generative fill UI. Check out the video on the landing page. Lmk if you face further issues, and sorry about the lacking instructions (I'm not a great webdev).

Cos it's immersive (and interactive). Check out this realtime demo of 3DGS in Unity by Aras P (co-founder of Unity): https://www.youtube.com/watch?v=0vS3yh908TU&ab_channel=ArasP...

I like the way you think ;)

Fill 3D takes a different step from diffusion, in that it tries to build an actual 3D scene (kinda like a clone) of what's in the image you upload. In some sense, that's actually the most fundamental representation of what's in your image (or said another way, your image is just a representation of that original scene).

So it works by trying to estimate a 3D 'room' that matches your image. Everything from the geometry, to the light fixtures, to the windows. It's heavily inspired by how humans (weird to contrast 'human' vs. AI work) do image/video compositing.

TL;DR: Image in, 3D scene out.

Yup this is actually a roadmap feature. Because we generate in 3D, users can bring their own 3D models and add it to the catalog. And if you add something like object capture from Apple (https://developer.apple.com/augmented-reality/object-capture...), you could literally scan your couch, upload it to Fill 3D, place, and generate.

Exciting times ahead.

Depends on the image. Right now, the very first stage (deprojecting the image to 3D) makes assumptions about the image having the structure of a room: large empty floor plan; roughly polygonal geometry.

For different kinds of images, it's a question of using other cues to build a 3D structure that's very close to the original image. And no, monocular depth estimation isn't enough (happy to nerd out about why) ;)

Excellent suggestion. Will find time tomorrow to add a `/gallery` page. Created an issue to track: https://github.com/fill3d/fill/issues/1 . Best first issue :D

I couldn't agree more! You should check out the amazing work from the folks at Luma Labs (https://lumalabs.ai/). They're a loose inspiration for this project.

What bugs did you encounter? And yes, because we're using actual 3D models, there's a fixed set of models (right now, just under 300). Because the priority is ultra-realism, the current state-of-the-art for 3D model diffusion won't cut it (see OpenAI Point-e https://github.com/openai/point-e).

Very curious to learn more, how can I reach you? Or, shoot me an email: yusuf@fill3d.ai

Thats correct! Right now, we're using the Blenderkit dialog, but we can expand beyond it. When you type a prompt and search though, that's actually doing a multi-modal search (so you can ask for a 'red painting' and it'll actually find a red painting), so it's insanely more accurate than a regular search. AI everywhere!