1. What's the advantage?
2. How do you manage the state of the environment?
The advantage is programming-as-teaching, as opposed to programming-as-carpentry. What I mean by "programming-as-teaching" is that old-fashioned Smalltalk (and Lisp) enviroments like Pharo are designed to support a style of development where you build a program by starting up the runtime and teaching it interactively, incrementally, how to be the program you want.
(What I mean by "programming-as-carpentry" is the more widespread and better-known model in which you are essentially working to build a planned artifact from a blueprint, and your development environment is analogous to a workbench or toolchest, rather than a student.)
If you don't like programming-as-teaching, then it's not an advantage. Most people don't work that way, but I think that's mostly because they don't even know it's an option. I'm not saying that everyone would prefer programming-as-teaching, but it gets so little exposure that I think it's a safe bet that more people would like it if they knew what it was.
I prefer programming-as-teaching over the alternative. I am dramatically more productive when I can work that way. I've been one of those fabled 10X programmers, as confirmed by VCS statistics, when working with such tools. It wasn't because I'm so brilliant or because I have amazing magical powers; it's because we were working with that kind of environment, and I knew how to use it and the other programmers didn't.
So, in brief, the advantage is that if you have such an environment and can use it effectively, you can be dramatically more productive. It changes the feel of programming from "let's build this variant and see how it turns out" to "reach over and correct that student's posture." The cycle of iterations becomes very fast, so fast that sometimes the duration of a cycle is imperceptible.
How do you manage the environment? First, let me acknowledge that modifying your working environment willy nilly is indeed a problem. The solution is that you don't do that.
Instead, you start from a known good system image. Environments like Pharo and other old-fashioned Smalltalk and Lisp environments offer a feature that can save the entire state of system memory to an "image" file. If you start the environment from an image, you get an in-memory state that is an exact duplicate of the state the environment was in when saved--right down to the positions and contents and activation states of all the windows. You start the image, and you're in the environment, exactly as it was when you saved it.
The way you manage incremental state changes is that you make a reference image that contains a known good state. A common choice for a reference image is a clean, unmodified release image. Then you make any site-specific modifications and customizations and save the result as a reference image as well. Then you save a working image based on that reference image and make your changes to the working image.
This practice enables you to easily recover if you get into a bad state. It's very fast to kill a bad session and restore a reference image. For example, a Pharo 6.1 image launches in under half a second on my older iMac. Clozure Common Lisp starts from an image in a similar amount of time.
When I've done ongoing serious work using this kind of system, we kept orderly collections of reference and working images--reference images that served as checkpoints in case of trouble; working images in which we had all the stuff we were working on loaded up and ready to go.
Besides making it quick and easy to resume yesterday's work, and to recover in cases where we messed up, images also served the useful purpose of enabling us to capture precise dynamic states of work in progress. For example, if a rare bug arises, I can save a working image with the problem on the screen, and give that image to my colleagues. They can start up the image and immediately see the problem state on their screens, and use the introspective capabilities of the development system to diagnose the problem.
But I guess that's more an advantage than a how-do-you-manage-it observation.