rudeshark.net/custom/assets/robots.txt

15 lines
236 B
Plaintext
Raw Permalink Normal View History

User-agent: *
Allow: /
# Uncomment the following to block CommonCrawl
#
# User-agent: CCBot
# User-agent: CCBot/2.0
# User-agent: CCBot/3.1
# Disallow: /
# Uncomment the following to block ChatGPT
#
# User-agent: GPTBot
# Disallow: /