Does anyone have suggestions on what a proper robots.txt would be?
How about:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlHow about:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlOtherwise that is the canonical minimal example.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap.xml
Won't crawlers crawl by default?