Monday, January 9, 2012

Blogger is restricting category/tag pages for search-engines

Google webmaster reported me crawl error, that my blog's http://ashish-thakur.blogspot.com robots.txt restrict 14 url to crawl by search engines.





Blogger said robots.txt is here http://www.yourblogname.blogspot.com/robots.txt. which looks like:

User-agent: *
Disallow: /search
Allow: /
 
So my all tag urls are not indexable.. damn. And google says you can not change your robots.txt http://www.google.com/support/forum/p/blogger/thread?tid=18ba1cea84692c28&hl=en

I feel like this is one of the reason, we should not use blogger for high end seo website.

No comments: