.png)
Few months back blogger has added new feature of adding Custom Robot.txt. This is very useful in setting the visibility of our articles on search engines.Using Custom Robot.txt helps us to control the indexing of the article by Search Engine.
By default you can see the robot.txt like shown below
Below is the Explanation of the Robot.txt, just have a look
Mediapartners-Google this is a robot from Google Adsense, Altering it may cause lead adverse effects in showing ads so take caution altering this.
User Agent : The next line is user agent it is for all the robots and marked with an asterisk (* means all). From the default configuration it is clear that the label of our blog is not indexed Disallow: /search.
This is to be noted that the slash (/) is our homepage, Now for example if you want that all the labels to get indexed, never just fill up a slash like this "Disallow: /" as that will not allow your blog to be crawled,It should be like shown below
Using the above configuration all of your article and label will be indexed . If you want to block any page say abc.html from your blog you have to use robot.txt as shown below
Sometimes this causes pagination problem . To resolve the pagination issue on blogger we remove the Disallow: /search than Our robot.txt will look like this :
User-agent: Mediapartners-Google
Disallow:
Now after doing all changes just verify every thing done is fine. Just visit Sitemap:http://www.websitename.com/feeds/posts/default?orderby=updated
for Checking your robot .txt where websitename is the name of your website
Caution !!! Use the above information with precaution. Incorrect use of these features can result in ignorance of your blog by the search engines.
By default you can see the robot.txt like shown below
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://www.website-name.com/feeds/posts/default?orderby=updatedBelow is the Explanation of the Robot.txt, just have a look
Mediapartners-Google this is a robot from Google Adsense, Altering it may cause lead adverse effects in showing ads so take caution altering this.
User Agent : The next line is user agent it is for all the robots and marked with an asterisk (* means all). From the default configuration it is clear that the label of our blog is not indexed Disallow: /search.
This is to be noted that the slash (/) is our homepage, Now for example if you want that all the labels to get indexed, never just fill up a slash like this "Disallow: /" as that will not allow your blog to be crawled,It should be like shown below
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow:
Allow: /
Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updatedUsing the above configuration all of your article and label will be indexed . If you want to block any page say abc.html from your blog you have to use robot.txt as shown below
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /p/faq.html
Allow: /
Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updatedSometimes this causes pagination problem . To resolve the pagination issue on blogger we remove the Disallow: /search than Our robot.txt will look like this :
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /
Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updatedNow after doing all changes just verify every thing done is fine. Just visit Sitemap:http://www.websitename.com/feeds/posts/default?orderby=updated
for Checking your robot .txt where websitename is the name of your website
Caution !!! Use the above information with precaution. Incorrect use of these features can result in ignorance of your blog by the search engines.
