The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, `Disallow: /help' disallows both `/help.html' and `/help/index.html', whereas `Disallow: /help/' would disallow `/help/index.html' but allow `/help.html'.
Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.
Go to the first, previous, next, last section, table of contents.