Set in server config or via your CDN. Critical for managing index status of file-based content.
It's the only way to noindex non-HTML resources like PDFs.