If you wish to inform the robots about the location of your sitemap then simply enter the sitemap URL below.
Example: "http://www.chameleonwebservices.co.uk/sitemap.xml"
Now click "Download" and save the robots.txt file to your local machine and then simply upload to the root of the domain using an FTP client or hosting account file editor.
Search Engine Robots are known as Web Crawlers or Spiders, and they are used to automatically travel across the Internet finding information from websites and following links from webpages to find more pages and then following more links.
Search engines such as Google use robots to index web content but robots can be used for negative operations also such as the use by spammers to crawl the Internet looking for email addresses.
In order to disallow or block search engine robots we use a robots.txt file which give us the ability control web robots using the "The Robots Exclusion Protocol".
To control the robots we set specific commands in the robots.txt file. When the robot enters a webpage or follows a link to a webpage the first thing that happens is it will check what is allowed and disallowed.
Example:-
User-agent: * Disallow: /
The User-agent: * applies to all robots.
The Disallow: / informs the robots to not visit any pages on the site.
Simply use the FREE robots.txt generator above to create your specific instruction set. We have pre-configured data for WordPress and OpenCart, simply click the button relating to the platforms
Chameleon provide this tool for FREE along with many other tools like our FREE SEO Website Checker which show you what a robot sees when it visits your website.
