Monday, January 9, 2012

Blogger is restricting category/tag pages for search-engines

Google webmaster reported me crawl error, that my blog's robots.txt restrict 14 url to crawl by search engines.

Blogger said robots.txt is here which looks like:

User-agent: *
Disallow: /search
Allow: /
So my all tag urls are not indexable.. damn. And google says you can not change your robots.txt

I feel like this is one of the reason, we should not use blogger for high end seo website.

No comments: