How To set up your Robots.txt file in wordpress

The Robots.txt file is a simple text file uploaded to your website to deliver instructions to search engine robots that crawl your website looking for content and pages to index. Having a Robot.txt file is a great SEO trick because it directs the robots to areas of your site that have ranking value based on Google’s own algorithm for what constitutes a good webpage.

Google is the largest search engine online today.

It keeps track of Billions of websites and details the content that is on each website, so that someone looking for that content can find it based upon the search term used. The problem is Google does not distinguish one webpage from another. so unless you specifically set some protocols to instruct the search engine spiders then they will crawl and index every page they find when they visit your blog or website. Google and all the other search engines use “Robots” to crawl web pages looking for unique content to index in its web page listings service. Websites listed are then eligible for discovery during a search.

Search Engines read a /Robots.txt file to get information on what they should and shouldn’t be looking for, and where to find stuff on your website.

Specifying where search engines should look for content in high-quality directories or files you can increase the ranking of your site, and is recommended by Google and the other major search engines at Yahoo and Bing.

What is a Robots.txt file

To offer some control over the search engine robots , Web site owners and blog site owners use a special file protocol that gives specific instructions about their site to web robots; this is called The Robots Exclusion Protocol. the protocol is a simple and is a text file called /Robots.txt file is just that, a simple text file that you write in any standard text editor such as Notepad in Windows or Text editor on a Mac.

It works likes this: a robot discovers a Web site URL, say http://www.mysite.com/welcome.html. Before it does so, it firsts checks for http://www.mysite.com/robots.txt, and hopefully it finds a txt file called /Robots.txt with the following statement written:

The “User-agent: *” means this section applies to all robots.

The “Disallow: /” tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

1/ Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters
used by spammers will pay no attention.
2/ The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

So don’t try to use /robots.txt to hide information.

Robots.txt – how it works

When a robot looks for the “/robots.txt” file for your URL, it strips the path component from the URL (everything from the first single slash), and puts “/robots.txt” in its place.

So using our earlier example , for “http://www.mysite.com/a folder/index.html, it will remove the “/a folder/index.html”, and replace it with “/robots.txt”, and will end up with “http://www.mysite.com/robots.txt”.

So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site’s main “index.html” welcome page. Where exactly that is, and how to put the file there, depends on your particular hosting account configuration.

Robots.txt -the wordpress solution

However in the case of a Blog site, the HTML file is part and parcel of the whole software suite, and much harder to figure out without some deep understanding of wordpress coding. So as in all other cases for wordpress someone has made a nice simple Plugin for the purpose. Robots txt plugin for wordpress is a simple ready to go out of the box solution for wordpress.

This simple plugin once installed and activated carries out this important task and comes pre-loaded with most of the code already in the plugin so you don’t have to worry about the small stuff. However there are some adjustments I recommend that you make to the robots.txt list of commands contained in the plugin.
Watch the above video for more details.

The Robots.txt file is an important part of the blog structure and its endorsed by Google so you should take notice of that fact. So set up your Robots.txt file NOW!!