Yes, exactly. The procedurally generated world is predictable* if you know the world seed -- land forms, structures and items/loot spawn the same way each time. You can specify the seed when creating a world, if you like.

There are certain things that are nondeterministic, such as where mobs spawn (that depends on player position, etc.) or how they behave.

* There may be parts of world generation these days that are nondeterministic even with a known seed; I'm not too up to date with recent world generation changes.