There are a lot of ways to do this.
Most web applications have a parameter in them that points to one or more database servers of some kind, so you can create a new instance by doing, say, "CREATE DATABASE" in mysql and installing the software in a directory on a web server configured to point to the database. If there is a need for scratch directories and other resources that can be handled pretty much the same way as the database.
The provisioning system then is completely separate from the main body of the app so it is easy to keep one private and open source the other.
With the above you can often host a huge number of instances on one machine without any exciting tech (I was doing this at least as early as 2003) but obviously the idea could be implemented with containers, cloud servers, etc. today, for instance you create a cloud server for the user and auto-provision the software to the server.
There are more fancy "multi-tenant" systems such as Salesforce.com where you put a "tenant id" on the database fields so that many tenants can run in a single database, but that's another whole issue in which case some clever thinking would be necessary to add multi-tenant to the open source software.