A hardware wallet is a physical device that stores some private keys in a tamper-proof secure element. Those private keys can be regenerated from a recovery phrase[1] which acts as a seed to regenerate the keys in a deterministic way.

The hardware device is typically itself secured by means of a pin. Without the pin, the device can’t be unlocked so can’t be used, too many incorrect pin attempts will brick the device.

So the answers to your questions are:

1) If you entrust it to a safe deposit box then if someone steals it, it is worthless without the pin.

2) If the safe is itself destroyed and with it the device (this is also the case if you have it in a safe deposit box and the depository is burned down or something) then the private keys (and transitively the funds) can still be recovered using the recovery phrase. So if you have securely stored your recovery phrase and are able to retrieve it even this kind of problem won’t cause the accounts to be lost.

So what people tend to recommend is choosing good secure storage for your pin, keeping reasonable physical care of the device, taking the recovery phrase and splitting it into parts and storing those parts separately. If one of the parts is destroyed then you will need to urgently replace the hardware wallet, move the funds and securely store the new recovery phrase because if not you don’t have a fallback if the hardware wallet is destroyed, but otherwise you are good.

[1] https://medium.com/coinmonks/mnemonic-generation-bip39-simpl...

So don’t you have to secure this recovery phrase as well as the hardware wallet?

So if someone doesn’t have my wallet but has my recovery phrase they can regenerate my keys and brick my hardware wallet as it sits in my home safe??

Yes you absolutely do[1]. But that’s true of any wallet (software wallets also have the exact same recovery phrase system so for example if you lost the hardware wallet you could configure a software wallet by using the recovery phrase and get your crypto back).

Someone else using your recovery phrase to steal your private keys wouldn’t actually brick your hardware wallet. It would still work but obviously since the thing that it was there to secure (your keys) had been stolen that would be moot.

The subtext is that keeping all this stuff secure is hard and depending on your threat model may not be worthwhile. This is similar to the way in which for most people it makes sense to have a bank look after their funds. In the world of crypto though we’ve seen obvious examples of these centralised custodians being untrustworthy and since they are not regulated or FDIC insured or anything of that kind it’s much more risky.

[1] If you want the ability to recover your funds if the hardware device becomes inoperable, lost, stolen etc. If not you could just burn the recovery phrase so you don’t need to secure it.

I wonder, couldn't a such "wallet" be built on top of secure element (i.e on iOS/android)? Carrying around an additional device just for "wallet" features is very inconvenient.

Presumably yes, although some people prefer having a special-purpose device even if it is an extra thing to carry around. It does depend on your threat model I guess.

Well, both of them are less likely to lose your money than bitcoin apparently.

Especially if you stored solid brick of gold instead of money