I've seen apps break in prod where third party libraries had array accesses on indices that didn't exist, or dereferenced nil pointers, in goroutines they spawned. And those aren't the only ways to cause a panic without ever calling the panic function.
Third party libraries have bugs sometimes, and other times our usage of them can create bugs. Nothing like a SIGKILL or anything malicious. That's just the way things are and will be. And breaking down like this in dev can be fantastic!
But when a once in a blue moon issue crops up in prod and all other connections are dropped, the fact that a simple defer+recover couldn't have been preemptively placed on all goroutines is a huge nightmare. I'd even take a "you can add defer+recover for any goroutines created further in this callstack" - it's rare that libraries spawn goroutines, I'll already know which ones to trigger that behavior on
> But when a once in a blue moon issue crops up in prod and all other connections are dropped, the fact that a simple defer+recover couldn't have been preemptively placed on their goroutines is a huge nightmare
Yeah, this is one of the weird things, to say it mildly, that Go does that is hardly mentioned anywhere. Most people don't know about it until they themselves first encounter the nightmare bug similar to what you said.
Wrapping every third-party library in defensive code doesn't make a lot of sense since this isn't the most dangerous kind of bug. To make a one example of a million: a third-party database library that corrupts your database is infinitely more dangerous and that there's no form of wrapping that will save you.
Bad code is bad code. Bugs are bugs. This issue is not a major source of problems in Go code.
Third party libraries have bugs sometimes, and other times our usage of them can create bugs. Nothing like a SIGKILL or anything malicious. That's just the way things are and will be. And breaking down like this in dev can be fantastic!
But when a once in a blue moon issue crops up in prod and all other connections are dropped, the fact that a simple defer+recover couldn't have been preemptively placed on all goroutines is a huge nightmare. I'd even take a "you can add defer+recover for any goroutines created further in this callstack" - it's rare that libraries spawn goroutines, I'll already know which ones to trigger that behavior on