If you're using wordpress or another pre-baked system, then it's worth having a sitemap plugin (example: Yoast SEO) or the built-in sitemap (like for Shopify).
Submit it to Google Search Console, Bing, and you're done (essentially).
https://support.google.com/webmasters/answer/7424835?hl=en#h...
> robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header.
If it's something like a staging site, you'd be better putting it behind a login screen to prevent mistakes anyway.
crawling != indexing
with a segmented sitemap (i.e.: one per pagetype, per namespace, ..) you get important communicated to indexed ratios you otherwise do not get.
if crawling and indexing works, it works but if it doesn't you should know it (see it in google search console) and debug it (narrow it down via segmented sitemaps).