Recently I learned that Discourse serves a totally different application to crawlers, like GoogleBot for example. I learned this when writing a Discourse plugin code to alter the meta <link rel="canonical"> tag based on if the topic was imported from the legacy forum or created new in the Discourse forum.
Much of this journey of discovery is chronicled here::
For the time being, I decided to simply strip the canonical tags from Discourse, regardless of whether the page is served to a user (where it is not needed anyway) or to a crawler.
I do this in the container yml file here:
run:
- exec: echo "Beginning of custom commands"
- exec: sed -i '/canonical_link_tag/d' /var/www/discourse/app/views/layouts/_head.html.erb
- exec: echo "End of custom commands"
Further confirmed in the source code that indeed this works as expected and all canonical tags are stripped out of the Discourse pages served to GoogleBot: