It is perhaps an opinion more than a fact, but an opinion that is likely widely shared by people who have done both. Computer assembly languages are not conceptually complex in the same way that higher level math is. The difficulty of assembly comes from its relative obscurity, the proliferation of instructions and architecture variants, and the process of analyzing a problem into the required steps. These are things that can be overcome with patience and reference manuals. At its heart, assembly instructions are doing simple things.

At the risk of an analogy, I would describe assembly programming as memory-intensive, and math as compute-intensive.