The default taxonomy view takes over the term display route, but links to it are not correctly evaluating access with the view's arguments.
If you have a taxonomy that should only show the label to anonymous users, you can set the validator to check for edit permissions instead of view.
That works to block access to the view, but the link still renders.
Issue module