The size of libsystemd is immaterial in the case of xz. The attackers had control of xz, and wanted to load it from sshd.

There's lots of projects that link xz, big and small. Patching sshd to include any of them would have implemented the backdoor.