robots.txt

看百度的 User-agent: BaiduspiderDisallow: /baiduDisallow: /s?Disallow: /ulink?Disallow: /link?User-agent: GooglebotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: MSNBotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Baiduspider-imageDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: YoudaoBotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou web spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou inst spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou spider2Disallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou blogDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou News SpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: Sogou Orion spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: JikeSpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: SosospiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: PangusoSpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User-agent: yisouspiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproDisallow: /ulink?Disallow: /link?User