Given the increased prices $0.70 for thie RP2040 seems like a good deal.
It'll still be <$2, but it also means putting another chip on your board and routing around it. The old ESP32s worked that way too, but most people used the SoM "cans" when those chips were new because of the complexity in routing an antenna trace.
Personally, I'll wait for gen2 and pray for a simpler bootstrapping process for bare metal code.
To be pedantic, there is a built-in 16kB ROM starting at address 0x00000000 with RPi's open-source BootROM burned into it at the time the silicon is manufactured. There's no publicly-known way to write to this. It's technically "non-volatile storage", but it's not useful to anyone other than RaspberryPi/TSMC.