I'm the total opposite. I'm so glad that when my phone get stolen or suddenly breaks (like my previous phone did), I can still drive my car, go home, make purchases, etc. and I'll only be locked out of services that (against my wishes) forced some phone based authentication.
There's no reason why you couldn't be offered both, access via your phone and a physical key/fob.
If access is granted digitally (rather than being statically issued via fob), then you could be able to grant people permission to enter dynamically. That way any number of devices could service the role of being a key.
You could use a phone, you could use a smart watch, you could use a QR code, you could use a smart fob or even something like a surgically embedded smart fob (into your hand).
Perhaps something as simple as SSH style private/public key exchange would suffice.
You could allow access to your apartment, car, workplace, gym through a web portal. Much like Github repository permission, match access to a person's identity. Perhaps you give some people free access while others you'd like to see a push notification for.
As far as security goes, while a fob is stealabe and cloneable, digital identities (think oauth-oidc) are significantly harder to obtain ilictly and it's also probably cheaper than the fob system.
