Support for multiple X-Robots-Tags

[ozboss]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
The Nextcloud Overview page issues a warning when the X-Robots-Tag is not set exclusively to 'none'.
The problem is 'none' is not supported by Bing. This causes multiple Nextcloud instances to be indexed by Bing, even though this is probably not intended.
One example would be https://cloud.dresden.de/. The page is easily found by searching in Bing and it does have the X-Robots-Tag set to 'none'.

Describe the solution you'd like
One solution would be to keep the requirement for the 'none' tag but allow more tags. This way X-Robots-Tag could be set to 'none,noindex,nofollow'. This way the instance should not be indexed by Bing as well as other search engines.

Describe alternatives you've considered
The alternative currently is to ignore the warning, which is kind of annoying.
(A way to disable warnings would also be appreciated.)

[ozboss]

Well this issue might have been created prematurely, after looking closer these pages also appear on the Google search results. But without any information ... just like on Bing. So apparently pages with a lot of access still appear in the search results.
Is the 'none' tag then supported by Bing? I found only information on it not being supported. But apparently the behavior is the same as on Google and it does not actually stop it from showing up in search results.

[kesselb]

I guess it's fine to change none to noindex,nofollow.

https://developers.google.com/search/docs/advanced/robots/robots_meta_tag
https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240

Feel free to submit a pr.

[MichaIng]

Wow, this explains a lot, never know that Bing does not support none. Yes makes much sense then to change it.

Btw, Nextcloud also sets the HTML meta tag

<meta name="robots" content="noindex, nofollow"/>

So actually one should be safe also for Bing. I'll open a report.

[MichaIng]

PR up: #36689

It does not exactly address the request here to support multiple values, i.e. for not unsetting but appending the values via .htaccess (?), but for a Nextcloud instance these are the only correct values I'd say, and for Google equivalent than before.