# 梦想直播官网爬虫规则 # https://www.mengxiangzhibo.com/robots.txt User-agent: * Allow: / Allow: /en/ Allow: /guide/ Allow: /wuren-zhibo/ Allow: /ai-zhibo/ Allow: /shuziren-zhibo/ Allow: /gongchang-zhibo/ Allow: /obs-zhibo/ Allow: /facts/ Allow: /pages/ Allow: /llms.txt Allow: /.well-known/llms.txt Allow: /ai.txt Allow: /sitemap.xml Disallow: /manage/ Disallow: /api/ Disallow: /pgweb/ Disallow: /download/*.exe Disallow: /download/*.7z # 国内常见爬虫 User-agent: Baiduspider Allow: / Disallow: /manage/ Disallow: /api/ Disallow: /pgweb/ User-agent: Sogou web spider Allow: / Disallow: /manage/ Disallow: /api/ User-agent: 360Spider Allow: / Disallow: /manage/ Disallow: /api/ # 生成式引擎 / AI 爬虫（允许读公开页与 llms.txt） User-agent: GPTBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: Google-Extended Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: anthropic-ai Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: ClaudeBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: Bytespider Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: CCBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ # 更多生成式引擎 / AI 搜索（GEO） User-agent: PerplexityBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: Amazonbot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: cohere-ai Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: KimiBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: DeepSeekBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: QwenBot Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ User-agent: BaiduAI Allow: / Allow: /llms.txt Disallow: /manage/ Disallow: /api/ Sitemap: https://www.mengxiangzhibo.com/sitemap.xml