# https://www.robotstxt.org/robotstxt.html User-agent: * Allow: /$ Allow: /how-to$ Allow: /privacy$ Allow: /sitemap.xml$ Disallow: * sitemap: https://crab.fit/sitemap.xml