Here's the suggested robots.txt file syntax with all the recommendations included: # ============================================================= # robots.txt for rampiq.agency # SEO + AI Search crawler configuration # ============================================================= User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Login and registration Disallow: /wp-login.php Disallow: /xmlrpc.php # Block internal search and low-value parameter URLs Disallow: /?s= Disallow: /search/ Disallow: /*?utm_= Disallow: /*?fbclid= Disallow: /*?gclid= Disallow: /*?msclkid= Disallow: /*?_gl= # RSS and comment feeds. Currently being crawled and consuming crawl budget Disallow: /feed/ Disallow: */feed/ Disallow: /comments/feed/ Disallow: */trackback/ Disallow: /*?replytocom= # ------------------------------------------------------------ # Keep public assets crawlable for rendering and indexing # ------------------------------------------------------------ # CSS, JavaScript and images must remain crawlable. Google needs # them to render pages; AI crawlers reading the rendered DOM need # them too. Allow: /wp-content/uploads/ Allow: /wp-content/themes/ Allow: /wp-content/plugins/ Allow: /*.css$ Allow: /*.js$ Allow: /*.png$ Allow: /*.jpg$ Allow: /*.jpeg$ Allow: /*.gif$ Allow: /*.webp$ Allow: /*.svg$ # XML sitemap Sitemap: https://rampiq.agency/sitemap_index.xml # ------------------------------------------------------------- # AI & LLM Bot Access # Explicitly allow crawling by AI search agents for discoverability # ------------------------------------------------------------- # Allow OpenAI bots User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Allow Perplexity bots User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Allow Anthropic bots User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Allow Meta bots User-agent: meta-externalagent Allow: / # Allow DeepSeek bots User-agent: DeepSeekBot Allow: / # Allow Google Gemini bots User-agent: Google-Extended Allow: / # Allow Bing bots User-agent: Bingbot Allow: / # Allow Apple Intelligence User-agent: Applebot-Extended Allow: / # Allow Common Crawl (widely used LLM training corpus) User-agent: CCBot Allow: /