It is for this reason that I use Arch Linux. Building a system from the ground up is, quite honestly, a wonderful experience (it does get tedious after four or five installs on different computers, but meh). I don't have any Crappy Crapware, or Auto Updates (sometimes this bites me), or Evil Bloat ... like a desktop environment. It's a breath of fresh air. As you say, there is no magic.