Setup robot.txt on blogger

Few months back blogger has added new feature of adding Custom Robot.txt. This is very useful in setting the visibility of our articles on search engines.Using Custom Robot.txt helps us to control the indexing of the article by Search Engine.


By default you can see the robot.txt like shown below                          

User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search 
Allow: / 

Sitemap: http://www.website-name.com/feeds/posts/default?orderby=updated

Below is the Explanation of the Robot.txt, just have a look

Mediapartners-Google this is a robot from Google Adsense, Altering it may cause lead adverse effects in showing ads so take caution altering this.

User Agent : The next line is user agent it is for all the robots and marked with an asterisk (* means all). From the default configuration it is clear that the label of our blog is not indexed Disallow: /search.

This is to be noted that the slash (/) is our homepage, Now for example if you want that all the labels to get indexed, never just fill up a slash like this "Disallow: /"  as that will not allow your blog to be crawled,It should be like shown below

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow:
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated

Using the above configuration all of your article and label will be indexed . If you want to block any page say abc.html from your blog  you have to use robot.txt as shown below

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /p/faq.html
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated

Sometimes this causes pagination problem . To resolve the pagination issue on blogger we remove the Disallow: /search than Our robot.txt will look like this :

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated

Now after doing all changes just verify every thing done is fine. Just visit Sitemap:http://www.websitename.com/feeds/posts/default?orderby=updated
for Checking your robot .txt where websitename is the name of your website

Caution !!! Use  the above information with precaution. Incorrect use of these features can result in ignorance of your blog by the search engines.