Fable 5.1 World Modeling | Hacker News Reader

Fable 5.1 World Modeling

github.com

327 pointssurreal_106 comments

I have experimented with this type of modeling for a RTS game I'm building: https://playwarpact.com

Few things:

1. Opus 5 is just as good for this tbh and cheaper.

2. They don't generate optimized 3d models. They have high poly count for simple geometries.

3. A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes of the 3d model and then bake textures that include a lot more details like windows, doors etc with tools like Meshy. Can post a tutorial if there's interest.

The models need some more RL to be able to do this autonomously.

"Can post a tutorial if there's interest."

There is.

Will write it up!

Thanks!

posted it here: https://news.ycombinator.com/item?id=49636173

Here, have another, Thank you!

https://news.ycombinator.com/item?id=49636173

Here's the tutorial - https://playwarpact.com/blog/lowpoly-hybrid/

why would you force me to login to see what the game is, I have zero information and you're asking me for a password or my google information. hostile and creepy.

Sorry, the game is still in dev and recently opened up for early access. I am working on a marketing page for the game that would link to the gameplay.

I’m interested! I struggled to get goos detail on my Back to the Future Delorean 3D animation on my docs hero: https://contextify.sh/docs/

I posted this previously and people wanted more info so here is how I built it, with a deeper dive into some film scene recreation I worked on focusing on a scene from Apocalypto:

https://banagale.com/cinematic-canvas-ai-film-animation.htm

I’d love to improve the look of these things.

Did you animate a 3d model of the car? or its a fully procedurally built animation?

It is a 3D-ish model being animated by a procedural motion system, rather than a pre-rendered/model animation.

The car started from a low-poly DeLorean glTF. I had the model rebuild the geometry into inline Canvas code rather than just dropping a model into three.js.

The flight itself is procedural: position/rotation, hover, pitch, wobble, flames, camera, etc. are all driven by code and can be scrubbed/debugged.

That link has a link to an interactive page where you can adjust the car’s path and when it reaches 88mph if you want to see.

It is not mobile friendly atm, though.

I would like to apply textures to the people and scenery in the gauntlet scene. Just haven’t had the time or tokens, as getting motion and pathing and camera angle right seemed the mvp.

meshy.ai , tripo are good ones to use. They give you free credits as well. If you already have the low poly model its easy to apply textures for it.

> Opus 5 is just as good for this tbh and cheaper.

It depends on the fidelity, there's a threshold slightly over "low poly" that Opus cannot get over. Once you get into creating foliage l-systems, or physical mob animation, or a house with realistic appliances, Opus is under the convergence threshold for a world model no matter how much time you give it, it will flail around and say it's done even though it's nowhere close to plausible. Fable takes forever but knows how to zoom in and out on the abstractions.

> They don't generate optimized 3d models. They have high poly count for simple geometries.

By default LLMs will do the quick prototype thing and slap some primitives into a THREE.js scene. Which is probably correct because most users don't know or care. But if prompted Fable will connect the manifolds/sculpt with a mesher, instance out the pieces, etc. and make you an efficient art pipeline. You can make it optimized. Just not in 10 minutes.

> A better approach I've utilized for game ready assets is to use the model to create low poly silhouettes

You can also let Meshy make the geo and then fit it back to the world with PCA + BB's. Diffusion models are still 10x better at making physically plausible geometries than LLMs, plus they are faster all things considered. One technique I've found that works well is socketing; let the LLM generate the high level structure with blockouts for sockets, then slot in the higher poly objects that fit. Which is closer to the professional approach for world design.

> then fit it back to the world with PCA + BB's

Can you expand on this?

I've tried several AI 3d generators that claim low-poly output, and they're all far off. None of them decide what belongs in the mesh vs. what should be baked in with the texture, so the poly budget lands in the wrong places.

This seemed to have been accidentally downvoted to death. Just vouched for it, as it's a normal comment

What I'm suggesting is if you want a certain semantic part of the scene to have a specific design or fidelity (say an interactive object, point of interest, character, or parts thereof), you can generate it via image and then use principal component analysis, bounding boxes, and a physics engine like Jolt to place and orient it programmatically into the right "socket" in your world so it aligns and joins correctly. The LLM can generate the high level spatial/semantic structure, and you can plug the assets into the right space. That works both at scene level and more locally (say a character's equipment they are wearing or holding).

This kind of decomposition helps to get the art composed the way you want it, though it won't help with a runtime poly budget.

But if you are intentionally doing low-poly style, it is doubtful you are (at least technologically) limited by having too many vertices on assets; you're limited by draws and submits and rendering architecture. You can push millions of animated polys with bells and whistles, postprocessing, physics etc, in the browser, at 90 FPS, on a macbook. You can have thousands of objects, but you have to cull/instance/share materials/atlas/etc, you can't just do `new THREE.Mesh` for each object -- which is what the LLMs will naively do by default unless you ask otherwise.

If resource size is a problem, meshes compress/quantize well with things like meshopt/DRACO.

oh, it's "play war pact".

I read it as "play, warp, act"

Warp Ship Politicians, warping around the galaxy enacting laws wherever they land. Could get some rather odd situations...

Kind of interesting that most ai games want me to sign in.

i know... i suffered through a long load but im not logging in without having even a semblance of clue what the game even is unfortunately

Regarding your game shared here: https://news.ycombinator.com/item?id=48943351

Uh, I tried feeding 5.6 Sol High my 2D DXF drawings of my house floor layout & house exterior to generate 3D visualization within blender via MCP... and it was miserable. No understanding on how windows attach to walls, misaligned bearing walls between floors, gaps all around the places, chimney not continuous, windows don't line up and are in wrong places, house parts swapped. And still struggling when I explain those things - that it is a house and real-life architectural constraints should be taken into account. But I just had to keep prompting to fix this, that, redo this and that and then it breaks other things until I fed up.

Part of it is probably struggle to understand the DXF and mixing up "wall lines" with "dimension/helper lines" (I tried also feeding PDFs)

Anyways, if someone has an idea how to improve 3D modelling with AI or how to make understand 2D drawings to understand rooms/walls/square meters, etc - would be nice.

DXF is just a tough format. Get sol to visualize it first and use GIS algorithms on it.

Are you passing well organized layers?

Please do post the tutorial. Also, your RTS looks great, I will definitely check it out!

thank you! working on it, will post soon.

Here's the tutorial - https://playwarpact.com/blog/lowpoly-hybrid/

Wait so Opus modeled your cool scifi buildings?

As you might have researched it, do you know if there's currently any AI model or arrangement of models that are able to produce truly game-ready 3D assets? Good meshes with appropriate poly count and good topology, decent UV maps and textures, etc. I've seen some products advertising that, but I'm not sure they are up to their promise, and I haven't had the bandwidth to research it in depth...

I created a skill just for the 3d models. Works with Unreal Engine too. It generates an image using stable diffusion, then it transforms the object in the image into a 3d object using Hunyuan3D-2. You can check it out here: https://github.com/LaurentiuGabriel/unreal-game-assets-creat...

> Hunyuan3D

I'd love to see a sample .fbx as a wireframe, because as far as I've tried that model and basically any other model under the roof that runs on less than 96GB VRAM and can spit out 3D, have had absolutely horrible outputs that are barely usable for anything else than lightweight prototype renders. I'd love to be proven wrong though, would help a ton with building higher fidelity prototypes of various things.

These are not game ready unfortunately. It will be clear that they are AI generated due to not so good mesh quality.

This is the kind of demo that would really benefit from a youtube demo of 2-3 min

Fake edit: there is a longer video here 1 min long: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...

I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo.

> I would be especially curious to see the NPC person/car logic and if they're on rails or what, that's a pretty good NPC density for a demo.

Here ya go: - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... - https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...

what's going on with these overlays? the readme is presenting those as though they are evidence of the accuracy of the project, but its completely unable to line up the shots

https://github.com/PhiloLabs/fable51-worlds/blob/main/union-...

AI generated everything. Dude probably didn't even look at the examples.

This isn't what I expected when I heard "world model"!

ik, "world model" has too many definitions at this point. imo the bar is just: can it simulate the world.

Not really. Whether you agree with LeCun's assertion that "LLMs Are a Dead End" or not, world modeling is the same domain. "World Modeling" in this post is confusing because it seems to reference the deep, specific machine learning that goes into these frontier world models and self-driving cars, where it's really just vibe coding turned at the problem of modeling the sort of 3D worlds you'd find in a video game.

Agreed. Author should not be using this term. A world model, as I understand, is what an LLM is missing that a human brain has: in addition to a linguistically described world, we also posses a sensory model of the world, a physics model so to speak, against which the random token/thought chains generated by our brains can be validated.

being reductive, causality is the key term for a world model. the rollout is the key action to demonstrate causality. lots of nice LLMs can generate a 3d representation of a "visual world". These days the focus is on sensory data (image, video, etc.), however world models can be more than just 3d spatial representations - they could be tabular. Again being reductive, its how do you predict the next state from the current state (the rollout).

These things always look impressive but I fail to see how they are usable beyond demos or simple games. In my experience the topology is often a mess and texturing is difficult as well

i discovered doing this in claude design and i can tell you... it's useful in a professional domain. Although I am not going to share what I'm using it for, to keep an edge in my domain ;)

Don’t worry I’m sure it’ll be in the training data soon enough…

Well, nothing now starts in a completed state. See pre-production footage for movies and video games, for how scenes progress.

Uncharted 4: https://youtu.be/NedDxIGQVs0&t=147

Coco: https://youtu.be/nl_JkjgHfFU&t=10

Up: https://www.youtube.com/watch?v=JHfLGgOs6gY

And, eventually you can just run it through Nvidia DLSS 7 for better hair physics and textures. ;)

Woah amazing clips thanks for sharing! The GH repo is definitely in early stages and prob laying the ground work for using the real world as part of the pre production process for a game/movie, etc.

I wish you would provide more information. About time, costs, reliability (how many duds did you have? did you need to do any babysitting or etc) and so on But it is a very impressive looking demo, for sure.

this is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop. ~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API

Did you need to iterate on the prompt, or did you have a model help you author it? I frequently have problems with orchestration instructions in-prompt, and your is huge. Maybe this is just better with Fable? I honestly haven’t used it much.

This is beautiful! I would love if we could model out a whole city, ideally using a much cheaper model.

I'm really curious how GML-5.3-flash would do. Very affordable, and it seems to do pretty well with 3D modeling.

yes experimenting with it actually, will update here! in fact we've generated most of the tourist spots in sf, should be reflected in the repo soon too

Hell yeah, whats your thoughts on doing it for like a whole city?

Nice! I had a similar positive experience treating OpenStreetMaps as the highest tier data source and then using screenshots to tune the aesthetics.

I wonder how high fidelity we can get these views using just ThreeJS and Fable5.1 iteration cycles.

Here is my Palisades Tahoe world that I made in a similar way: https://ski-red-dog-face.vercel.app/

Good work! I'm curious about how much token consumed to have this... Would it be a business secret?

I don’t really care about the real world, I want someone to make a thing where you take scenes from movies and shows and can then walk around in those worlds constructed by AI, to very high fidelity, and maybe in VR. Someone go do this, I’ll be on the lookout for the hackernews article.

Cool demo! That's funny. I literally sketched out something similar 30 minutes ago on paper. I wonder if there was some recent, semi-related article/link that subconsciously prompted the relevance.

we're working on a survey paper about world modeling via code, with folks from frontier labs (qwen omni, oai, etc) and academic institutions (e.g., oxford, stanford, etc) reach out to us about collab: team@philolabs.ai

This, maybe?

https://news.ycombinator.com/item?id=49525160

This reminds me of the experience VRML wanted to be 3 decades ago. The only difference is that links went to other worlds living on another server. But that seems trivial to implement with this.

VRML was incredible. It's a "hypertext" markup language but instead of documents it produces 3D environments. It's a shame it never really caught on.

There's some work going on at the Metaverse Standards Forum to progress these ideas with Web of Worlds (https://webofworlds.github.io/). It's scope is a lot broader than VRML, it covers assets, identity, avatars, payments, etc.

There's a link to the presentation at SIGGRAPH - https://youtu.be/N2_lb77gKQ8.

I know a guy in LA who basically spent every waking minute of his life past year or two building a spec for this. Announced soon apparently. I’ve spent hours reading some of it.

Pretty damn compelling.

A little late I know, but yes it was. The problem was the hardware didn't really support what VRML could do at the time. I had access to an SGI irix workstation and VRML (and 3d scenes) was still challenging for it back in the early to mid 1990s.

This demo is based on the wonderful (and actually hand-crafted) Three.js. It is a beautiful demo. The model expounded on the prompt and produced a very convincing web product based on Three.js. Wonder why wouldn't author try to produce this in plain WebGL... my guess - it is much harder and costly to do so.

So, I wonder when is Anthropic (or OpenAI (or... NAME_IT)) going to release one single stable super-working library that does... just anything, that we can then reuse the way these guys demonstrate somebody model's caps, while actually standing on the shoulder of giants.

Because they do stand on the shoulder of gigantic work done by Three.js team. Same goes for demos based on D3, imgui, etc.

This is really cool, but $33 for a single generated world makes it hard to see this being useful for games just yet. Not to mention how this would go in a much larger project

$33 is expensive? Compared to what?

There are free open source tools that convert osm data into 3D models, for example https://osm2world.org/. Also you could use freely available national 3d point cloud data, at least for some areas. both are $0 options. btw I am not saying $33 dollars here is expensive; just giving you some comparisons

Cool. But the question remains, what is the point of this? I don't mean to pick on this project specifically, but it's indicative of AI usage in general. You end up with something that's like "wow an AI made this" but it's not really anything that interesting or useful in and of itself. I've yet to see a single agent generated piece of software above that bar.

Because it’s a very complex task to do especially with any level of quality and consistency.

Classical algorithmic approaches for data to CAD are quite unreliable heck commercial CAD software still struggles turning scanned 2D plans into sketches with any degree of reliability.

Something of this scale would normally require 1000’s and 1000’s of man hours in manual modeling, placement and quality control.

Some of the latest prompt to CAD agents and models I’ve seen are the most impressive use of AI I’ve seen in a long time.

At least for me this is rather exiting since it gives me the ability to turn many more ideas into reality as a hobbyist rather than spending the entire weekend in Fusion 360.

I'm interested in this, what is the best current method/tool for this task?

Why don't you try to do it yourself with Fable and compare it with this one? Even with AI there are still a lot of work, knowledge, and gotchas involved. What's the point of such comment?

> I've yet to see a single agent generated piece of software above that bar.

Begs the question, an earnest one: what software would be above that bar?

the topology/texturing critique is fair for mesh generation, but code-generated worlds mostly sidestep it. when the model writes three.js or blender scripts, geometry comes from primitives, csg, and parametric construction, so topology is clean by construction rather than something you clean up after. texturing is still a gap, agreed, though procedural materials cover more than people expect.

Be nice to turn some of these into Counter Strike maps... mhmm...

Making callouts by SF street intersections, lol

I am making a spiritual successor the Rainbow Six Rogue Spear

Opus has been great at building the game engine but it does struggle with world building for me.

Any tips welcomed!

added the hand-painted style kyoto: https://github.com/PhiloLabs/fable51-worlds/tree/main/kyoto-...

this is really neat. I wonder if people will start building open world games or "AR" games in the vein of Pokemon Go and Ingress based on similar tech. I've had a similar idea for a long time but don't think it was feasible before now due to AI

yes we're working on it! trying to push a few open world rpg games with real economy and game design. it's also super interesting to benchmark the current models' capability in this direction, since this is a naturally hard and multimodal coding task

I'm not sure about expanding domains, but I expect that very close to 100% of games are now using LLMs to a very significant degree during development, even if most won't admit it. The "AI disclaimers" for games need to go away, simply because it's just rewarding and incentivizing lying and punishing truth.

Did anyone compare it with gpt-6?

yes we added the side by side comparison, check out the readme!

this is not a world model

Started the Matrix

simulate the universes and then observe the parallel universes

This looks great.

Very good

“Fable 5.1 World Modeling” is an incredibly misleading title! This has very little to do with Fable and using an unexpected sense of “world model”

fable 5.1 generated an interactive 3D union square, and the agent filmed its own tour guide vid inside it. you can walk Powell to Stockton, read the actual storefronts, cross a working intersection, watch a cable car go by. went inside Apple and the Nintendo store, lower level included

check source code + more worlds soon

Had to vouch for your comment, for some reason it was not visible.

> Geometry is derived from OpenStreetMap (ODbL) and USGS 3DEP (public domain).

This is probably how I would have tackled this, saves on time, especially if Fable just writes code to convert OSM data to a reasonable ThreeJS model.

Do you have data on the tokens or $ cost of this?

this is a one-shot result but i have a really really lengthy prompt: https://github.com/PhiloLabs/fable51-worlds/blob/main/union-... with clear guidance in using subagents and self-QA loop.

~2 hour (extensive subagents usage), total ~8M tokens, ~$33 under API

I shitpost on this website constantly and trash Anslopic on a daily basis.

I am your biggest hater.

But god dammit if I’m not also your biggest respecter.

reaches out for a handshake

You guys are onto something with this one.

Keep going.

You crazy bastards!

You crazy fuckin bastards you hear me?! Hahahahah

WOO!

Y U NO github pages?

Or where can I explore the demo app? It seems like it's just a static page!

All you need to do is create the gh-pages branch! And then you have it at https://philolabs.github.io/fable51-worlds/kyoto-higashiyama

Fundamentally it’s an image model predicting out frames of a sequence.

It’s not enough to call it just an image model though. Not quite video either because it’s so specific to rendering images of the world.

Thus, “world model”?

The problem I have with “world” is it should imply so much more depth. To me this is actually a “POV image model” or “First-person perspective model” (maybe we call this an FPM).

There could also be TPMs (third-person models).

This maps better to modern game and 3D dev phrasing, where “world model” might imply 3D object collections spatially organized in a cohesive file format - complete with interactivity, audio, and all the baseline constituents of what one might call a “world”.

FPMs and TPMs, think about it. Reserve POVMs for photographic realism.

I think perhaps you've misunderstood what the linked repo actually is? It seems to indeed be a 3d model of an area. It's not an image model, and nothing to do with frames.

You are correct - this is threejs and not video/generative rendering which I’ve seen a lot of.

I guess “world” works if interactivity and sound etc. are considered other modalities.

Video kinda has the same issue (for me) where the audio part of the video is not part of the model.

When I think “world” I think more than just the visuals. Like a “film” model to me would imply sound and more than just the video.

We typically call 3D art “models” too, so I guess it can’t be called a model model, idk the word “world” doesn’t seem right to me

Reply on news.ycombinator.com