To an extent, but only if the software meets the criteria for being legally classified as a "medical device". And at the lower classification levels the regulations are very light. You don't necessarily have to prove any level of reliability or usability.
https://www.fda.gov/medical-devices/classify-your-medical-de...