# ============================================================== # Shopify Robots Policy — Optimized for SEO + AI Crawlers # Last updated: 2026-11-22 # ============================================================== User-agent: * Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /checkout Disallow: /checkouts/ Disallow: /orders Disallow: /account Disallow: /search Disallow: /sf_private_access_tokens Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*?*oseid=* Disallow: /*?*ls=*&ls=* Allow: / # ============================================================== # AI & LLM Crawlers (Explicit Allow) # ============================================================== User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: CCBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / # ============================================================== # SEO / Analytics Bots (with rate limits) # ============================================================== User-agent: AhrefsBot Crawl-delay: 10 Allow: / User-agent: AhrefsSiteAudit Crawl-delay: 10 Allow: / User-agent: MJ12bot Crawl-delay: 10 Allow: / User-agent: Pinterest Crawl-delay: 1 Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # ============================================================== # Common Crawl and Archive Bots # ============================================================== User-agent: CCBot Allow: / User-agent: CommonCrawl Allow: / User-agent: InternetArchive Allow: / # ============================================================== # Explicitly disallow rogue or irrelevant crawlers # ============================================================== User-agent: Nutch Disallow: / # ============================================================== # Sitemap # ============================================================== Sitemap: https://topfoxx.com/sitemap.xml