Pages

Thursday, February 14, 2013

Setup robot.txt on blogger

Posted at  7:10 AM  |  in  blogging

Few months back blogger has added new feature of adding Custom Robot.txt. This is very useful in setting the visibility of our articles on search engines.Using Custom Robot.txt helps us to control the indexing of the article by Search Engine.


By default you can see the robot.txt like shown below                          

User-agent: Mediapartners-Google 
Disallow: 

User-agent: * 
Disallow: /search 
Allow: / 

Sitemap: http://www.website-name.com/feeds/posts/default?orderby=updated

Below is the Explanation of the Robot.txt, just have a look

Mediapartners-Google this is a robot from Google Adsense, Altering it may cause lead adverse effects in showing ads so take caution altering this.

User Agent : The next line is user agent it is for all the robots and marked with an asterisk (* means all). From the default configuration it is clear that the label of our blog is not indexed Disallow: /search.

This is to be noted that the slash (/) is our homepage, Now for example if you want that all the labels to get indexed, never just fill up a slash like this "Disallow: /"  as that will not allow your blog to be crawled,It should be like shown below

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow:
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated


Using the above configuration all of your article and label will be indexed . If you want to block any page say abc.html from your blog  you have to use robot.txt as shown below

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /p/faq.html
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated


Sometimes this causes pagination problem . To resolve the pagination issue on blogger we remove the Disallow: /search than Our robot.txt will look like this :

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /

Sitemap: http://www.websitename.com/feeds/posts/default?orderby=updated


Now after doing all changes just verify every thing done is fine. Just visit Sitemap:http://www.websitename.com/feeds/posts/default?orderby=updated
for Checking your robot .txt where websitename is the name of your website

Caution !!! Use  the above information with precaution. Incorrect use of these features can result in ignorance of your blog by the search engines.

Share this post

About Imman Vel

I am a professional Hacker and I Give you hacked and cracked stuffs for free.My Another WebSite..Follow me on Google+.Find me on Facebook.

1 comment:

  1. We're a gaggle of volunteers and opening a new scheme in our community. Your site offered us with valuable information to paintings on. You have performed an impressive task and our entire community will likely be thankful to you.

    ReplyDelete

About-Privacy Policy-Contact us
Copyright © 2013 IMMAN STUFFS. Blogger Template by BloggerTheme9
Proudly Powered by Blogger.
back to top
Chrome Pointer