mirror of
https://git.kescher.at/CatCatNya/catstodon.git
synced 2024-11-22 19:58:07 +01:00
a5992e5883
* Revert "Change robots.txt to exclude some URLs (#10037)"
This reverts commit 80161f4351.
* Let's block media_proxy
/media_proxy/ is a dynamic route used for requesting uncached media, so it's
probably bad to let crawlers use it
* misleading comment
4 lines
137 B
Text
4 lines
137 B
Text
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
|
|
|
|
User-agent: *
|
|
Disallow: /media_proxy/
|