For solving long term tasks like finding things that aren't there, you can turn the annotated scene into a templated description and feed it to a large-enough model trained on interactive fiction.

You are standing in a kitchen. Ahead of you to your right there is a large refrigerator with the handle on the right side. There is a set of cabinets to your left with a plate sitting on the counter above them.

> get beer

You don't see any beer here.

<< COT: I know that beer is often found in the fridge. I should try opening the refrigerator

> open fridge

Opening the refrigerator reveals 4 cans of beer.

> get beer

taken

Obviously we're still several years from this working, but it's very exciting to consider. Interactive Fiction narrative fed by real sensors plus chain-of-thought blocks as internal monologue.