Yes you absolutely do[1]. But that’s true of any wallet (software wallets also have the exact same recovery phrase system so for example if you lost the hardware wallet you could configure a software wallet by using the recovery phrase and get your crypto back).

Someone else using your recovery phrase to steal your private keys wouldn’t actually brick your hardware wallet. It would still work but obviously since the thing that it was there to secure (your keys) had been stolen that would be moot.

The subtext is that keeping all this stuff secure is hard and depending on your threat model may not be worthwhile. This is similar to the way in which for most people it makes sense to have a bank look after their funds. In the world of crypto though we’ve seen obvious examples of these centralised custodians being untrustworthy and since they are not regulated or FDIC insured or anything of that kind it’s much more risky.

[1] If you want the ability to recover your funds if the hardware device becomes inoperable, lost, stolen etc. If not you could just burn the recovery phrase so you don’t need to secure it.