Experience with SPM might vary depending on how many dependencies you’ve got. A lot of Apple platform apps are quite thin on third party library use, in which case SPM is probably not a large source of trouble.
As your app scales, you're going to want to break out different parts into their own packages in order to keep compile times in check, so you're gonna be forced to go through SPM unless you use tuist or bazel.
My problem with it is that I want to use C libraries. And I would (like) it to handle that as much as possible. But SPM can't use vcpkg packages or call CMake stuff (at least, not trivially), so it's extremely hard to use on non-Apple platforms. Which honestly is it's killer for me. It's still so, so Apple focused.
What is a CMake-built swift package to begin with? You're mixing build systems and expecting them to co-exist or what is the exact problem? I've done a lot of weird swift things so might be able to point you in the right direction.
E.g.: referencing a vcpkg-built package (without pkgconfig because not all packages have those files). Or telling SPM "Hey, I have this package which uses the cmake build system, and I want you to link to it and auto-generate module maps for it, and get the include directories from cmake". Things like that. So for me anyway it makes using swift painful. The same thing goes in reverse: using SPM packages from cmake (although this is more a cmake issue).