Hey,
I'm trying to block this as per the recommendation of an SEO check. I used this article http://forums.asp.net/t/1970637.aspx?I+want+to+block+libwww+perl but it doesn't seem to work me:
User-agent: libwww-perl Disallow: / User-agent: * Disallow: /_archive/ Disallow: /App_Code/ Disallow: /webservices/ Sitemap: http://www.domainname.com/sitemap.xml Host: www.domainname.com
What am I doing wrong? Is it case-sensitive (Libwww-perl); do the user agents need to go in a specific order?
Thanks