Robots.txt file is the one of the most important thing to take care of if you are performing SEO of your wordpress website or blog. Robots.txt file is a text based file which helps you to decide which part of your website you want to be crawlable by search engine bots. Wrong configuration of your robots.txt file can harm your SEO and your website could became invisible on search engine results.
Here in this article we will look after the common concepts of robots.txt file and how to set it up for better results. You can apply the stuffs we are going to discuss on your website and make it more SEO friendly for search engine bots.
What is robots.txt ?
If you are a beginner blogger or not familiar with web development concepts then probably you won’t know what is robots.txt file. Robots.txt file is a text based small file which resides in the root of your website. Your website could have or not have a robots.txt file. if you don’t have this file in root of your website then you can create it either from your hosting control panel or you can upload it through FTP tools like Filezilla. Here is blogithon’s Robots txt file and you can see the content and it’s the location at the root of the domain. EX:
http://www.blogithon.com/robots.txt
Configuring robots.txt File
After you have created the robots.txt file or if it’s already created in the root folder of your website, it’s time to configure this file for optimization and getting better results. We can setup rules for every search engine bots or spiders. We can also setup rules for individual search engines. We set this up with help of User-agent rule. After that we use Allow and disallow rules to allow or disallow indexing of our files or folders respectively.Adding the sitemap rule helps search engine to find your sitemap file and that result in faster indexing of your pages.
For Example:
User-agent: *
Disallow : /images
Now this rule will allow all search engines to crawl and index the whole website except the files and folders inside the images folder.
Here i am showing a sample robots.txt file for a wordpress website or blog. You can change it according to your needs if you want. Although it will work perfectly for any wordpress blog.
User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /archives/ disallow: /*?* Disallow: *?replytocom Disallow: /wp-* Disallow: /comments/feed/ User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Mediapartners-Google* Allow: / User-agent: Adsbot-Google Allow: / sitemap: http://www.blogithon.com/sitemap.xml
Now you have got the clear idea of importance of role of robots.txt file in SEO of your website. If you are facing any problem in optimizing your site or configuring your robots.txt file, let us know via comments. Also subscribe to our newsletter to keep yourself updated with new tips and articles.