Robots.txt – file.

Robots.txt tells good search engines which pages are to be included and which to ignore and not index. For example the simple lines:

User-agent: *
Disallow: /

In a text file called robots.txt in your WordPress root folder will stop your site being included in indices like Google.

The following however is good for WordPress:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

This tells engines to index the site but nor the two folders you can see. These two shouold be kept private. SImply make up a fiule and save it in your website’s root folder in a file called robots.txt.

Robots.txt as a WordPress setting

Edit your WordPress-based robots.txt file from the WordPress Admin using a WordPress Website plugin such as:

Christopher Davis’s, WP Robots Txt.

It’s  a plugin that adds an additional field to the “Reading” admin page.  With this installed and activated,  simply go into the WordPress menu option for website settings: Settings/Readings and scroll to the bottom and change information in the window below

Robots.txt Content