Managing robots.txt
Members of the System Administrators organization can manage the contents of the
robots.txt file for the Titania Delivery system as a whole. While each
portal can specify whether its path is included with an Allow or
Disallow rule, System Administrators can manually author the first part of
the dynamically-generated robots.txt file.To edit robots.txt, simply select the robots.txt
link in the header.
This link will open the Manage robots.txt page. The top of this page
is a simple text editor with save, revert, undo, and redo actions. Users can enter
the data
for the robots.txt content using this editor.
The bottom part of the screen displays the automatically-generated code based on the various portals' configuration settings.