I grew up near Oslo. There used to forests between where I lived and the nearby tiny town and between that town and the next one, before you got close to continuous urban settlements near Oslo. My brother was in the scouts. I still tease him about the time they got lost in the forest for 6-8 hours before they found their way out near one of the suburbs of Oslo.
Yet we still had perfectly functioning public transit.
Sure, there are definitive advantages that arise over time once you already have it, such as clustering of shops near transit hubs instead of spread out all over the place, but it still works.
It boils down to willingness to invest. You need to commit to ensuring sufficient frequency and routes for years to change habits even if it operates at a loss.
This is the real problem. You need to be prepared to put the service in before the demand is sufficient to really justify it, because the long term social impact justifies it. Do that, and it shapes the landscape. I mean, the train did that in the US too when it first arrived: entire towns sprung up around access to transit. The convenience of cars would make it harder to cause a reversal, but it's just a matter of which benefits you need to create.
So it's down to political will and lack of public support. The density is a red herring.