You could probably run a QDR+QSFP Infiniband card at around 32Gbps (minus overhead) through an external "GPU" enclosure. I don't see why MPI wouldn't work on Asahi Linux with such a setup once there's Thunderbolt support.

QDR Infiniband is, like, 2007 level tech. Today we have NDR Infiniband where a typical 4xHCA gets you 400 GB/s. Seems like a hypothetical Mac cluster would be severely limited by this compared to the typical x86 based server clusters.

I’m sure such a switch would be serving some pretty beefy nodes, though, right? Maybe the compute:communication can be held constant with less-powerful Mac mini nodes?

The problem is that the Thunderbolt ports don't support that bandwidth, and there isn't any other way to connect PCIe peripherals.

It is apparently possible to do networking over some Thunderbolt interfaces, would it be possible to connect the devices over Thunderbolt to one another directly? Four ports each, so form a mesh! I guess TB4 can go up to 40Gbps, although it sounds like there’s a bit of overhead when using it as a network, and also I have no idea if there’s some hub-like bottleneck inside the chip…