What is a Robot.txt file?

The “robots.txt” file is one of the most important files on your WordPress website, but it is one that will never be seen by your visitors.

Unlike most of the other files used in WordPress, robots.txt is not written in the PHP programming language. Instead, robots.txt is a very small plaintext document that contains a list of instructions for so-called “web crawler” automated scripts, otherwise known as “robots” or “bots.”

The way that Google and other search engines find and categorize your website is by using a web crawler bot (an automated script). The first thing the web crawler bot will do is read the robots.txt file on your website for instructions. These instructions may include things like telling the web crawler bot not to index your site at all in order to keep it private. Other instructions may include ignoring certain directories or file in order to keep them private or because they are irrelevant to the rest of the content on your site.

Note: If your website has subdomains, each subdomain must also have its very own separate robots.txt file.

If you’re new to WordPress, you don’t need to worry about modifying or creating a robots.txt file as WordPress will do that for you. If you make changes to your website such as classifying it as private, WordPress will automatically rewrite your robots.txt file to contain this instruction.

However, it is important to remember that not all web crawler bots are friendly and respectful of the instructions contained in the robots.txt file. Some web crawler bots will ignore the instructions and publish links to files or directories that you don’t want listed while other malicious web crawler bots will use the robots.txt file to search out certain files.