Thursday, 21 July 2016

In one amongst my previous posts, I had mentioned Custom Robots Header Tags Settings for blogger. If you've got browse that post then I hope you guys ar aware with its importance in search rankings.

Today, i'm back with a awfully helpful and should aware blogging term that's Robots.txt.

In blogger it's referred to as Custom Robots.txt meaning currently you'll customise this file consistent with your selection. In today’s tutorial, we are going to discuss this term in deep and are available to grasp concerning its use and advantages. i will be able to additionally tell you ways to feature custom robots.txt go in blogger.

So let begin the tutorial.

What is Robots.txt?
Robots.txt may be a document that contains few lines of easy code. it's saved on the web site or journal’s server that instruct the net crawlers to a way to index and crawl your blog within the search results. meaning you'll prohibit any online page on your journal from internet crawlers in order that it can’t get indexed in search engines like your journal labels page, your demo page or the other pages that don't seem to be as necessary to induce indexed. forever bear in mind that search crawlers scan the robots.txt file before locomotion any online page.

Each journal hosted on blogger have its default robots.txt file that are some things seem like this:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

Explanation
This code is split into 3 sections. Let’s initial study every of them subsequently we are going to find out how to feature custom robots.txt go in blogspot blogs.

User-agent: Mediapartners-Google
This code is for Google Adsense robots that facilitate them to serve higher ads on your journal. Either you're mistreatment Google Adsense on your journal or not merely leave it because it is.
User-agent: *
This is for all robots marked with asterisk (*). In default settings our blog’s labels links ar restricted to indexed by search crawlers meaning the net crawlers won't index our labels page links as a result of below code.
Disallow: /search

That means the links having keyword search simply when the name are unnoticed. See below example that may be a link of label page named SEO.

http://www.bloggertipstricks.com/search/label/SEO

And if we have a tendency to take away Disallow: /search from the higher than code then crawlers can access our entire journal to index and crawl all of its content and websites.

Here Allow: / refers to the Homepage meaning internet crawlers will crawl and index our blog’s homepage.

Disallow explicit Post
Now suppose if we wish to exclude a selected post from classification then we will add below lines within the code.

Disallow: /yyyy/mm/post-url.html

Here yyyy and millimetre refers to the commercial enterprise year and month of the post severally. let's say if we've printed a post in year 2013 in month of March then we've to use below format.
Disallow: /2013/03/post-url.html

To make this task simple, you'll merely copy the post computer address and take away the journal name from the start.

Disallow explicit Page

If we want to compel a selected page then we will use an equivalent methodology as higher than. merely copy the page computer address and take away journal address from it which is able to one thing seem like this:
Disallow: /p/page-url.html

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
This code refers to the sitemap of our journal. By adding sitemap link here we have a tendency to ar merely optimizing our blog’s locomotion rate. suggests that whenever the net crawlers scan our robots.txt file they'll notice a path to our sitemap wherever all the links of our printed posts gift. internet crawlers can notice it simple to crawl all of our posts. Hence, there ar higher probabilities that internet crawlers crawl all of our journal posts while not ignoring one one.
Note: This sitemap can solely tell the net crawlers concerning the recent twenty five posts. If you wish to extend the quantity of link in your sitemap then replace default sitemap with below one. it'll work for initial five hundred recent posts.
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you've got over five hundred printed posts in your journal then you'll use 2 sitemaps like below:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

Adding Custom Robots.Txt to Blogger
Now the most a part of this tutorial is a way to add custom robots.txt in blogger. therefore below ar steps to feature it.

Go to your journalger blog.
Navigate to Settings >> Search Preferences ›› Crawlers and classification ›› Custom robots.txt ›› Edit ›› affirmative
Now paste your robots.txt file code within the box.
Click on Save Changes button.
You are done!