名创801
|
本文以云主机上安装了windows server 2003系统为例，讲述如何拦截搜索引擎蜘蛛抓取页面，下面列出规则文件httpd.conf #Block spider
RewriteCond %{HTTP_USER_AGENT} (Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu) [NC]
RewriteRule !(^/robots.txt$) - [F] 注：规则中默认屏蔽部分不明蜘蛛，要屏蔽其他蜘蛛按规则添加即可 |
| 由中国名创www.chuang.top 整理发布。 |
