nginx中禁止屏蔽网络爬虫

if ($http_user_agent ~* "Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|ia_archiver|Tomato Bot") {
   return 403;
}
可以用 curl 测试一下
curl -I -A "qihoobot" www.xxx.com

You may also like...