# Southwest Weed & Pest Solutions - robots.txt # https://swpestweeds.com # Default: allow all standard search engines User-agent: * Allow: / Disallow: /admin/ # ----- AI Assistant Crawlers (allowed) ----- # OpenAI / ChatGPT User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # Anthropic / Claude User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Google AI / Gemini User-agent: Google-Extended Allow: / # Apple Intelligence User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # Common Crawl (used by many AI training datasets) User-agent: CCBot Allow: / # Microsoft / Bing AI User-agent: bingbot Allow: / # Meta AI User-agent: FacebookBot Allow: / User-agent: Meta-ExternalAgent Allow: / # Amazon User-agent: Amazonbot Allow: / # DuckDuckGo User-agent: DuckAssistBot Allow: / # ----- Sitemap & AI discovery ----- Sitemap: https://swpestweeds.com/sitemap.xml Sitemap: https://swpestweeds.com/sitemap-core.xml Sitemap: https://swpestweeds.com/sitemap-services.xml Sitemap: https://swpestweeds.com/sitemap-locations.xml # AI/LLM context resources (ingested by /llms.txt-aware crawlers) # These are indexed in sitemap.xml and referenced from /llms.txt: # /llms.txt - business summary, services, areas, AI guidance (llmstxt.org convention) # /llm.txt - same content, legacy filename (kept for back-compat) # /pricing.md - machine-readable service pricing guide # /locations.md - full directory of 108 location-service pages # /services-detailed.md - technical treatment protocols and methodology # ----- Crawl politeness ----- Crawl-delay: 1