"Game" has two meanings, and they're both very explicit. One comes from "game theory"—a game is a triple of a set of players, a set of moves, and a function mapping (player, move) tuples to payoffs. The other comes from "fun theory"—a game is a subset of human experience which is optimal in providing choices that are on the edge of being predicted by our instincts, allowing us to get into "the zone" as we follow different paths to try and find rewards.
Life itself conforms to the second definition—our instincts evolved specifically so that life would be the "best" game; games are, thus, microcosms or reimplementations of life (see, for example, The Sims, or WoW, but even an RPG with inventory management will reflect this clearly.)
I think at a minimum, a game needs some kind of structure, and that the consequences of the game - the outcome - isn't particularly important. The structure is necessary to prevent people from going too far - "fair play" - and the outcome needs to be relatively irrelevant - "it's only a game" - so that people don't take it too seriously, precisely so they can have fun.
I don't think either of these things is true about life.
I think games are a human formalization of the animal notion of play, which is about developing reactions and experiences in a safe way, unlike how they will be used for real stakes as an adult. Play fighting isn't real fighting, but uses many of the same reactions and develops techniques.
I'd like you to, then, define what is and isn't considered art, a process to which your exact argument — "everything isn't [art|a game]" — has been applied countless times (see: Duchamp's Readymades).
You could argue that many aspects of life are and always have been games; I think fnid2 has a valid point.