If someone else in the "exact same circumstances and starting conditions" implies they're identical down to every single molecule, how is that someone else?
If they're not identical at that level, they wouldn't make the same decisions. Put two almost-perfect clones into two exact copies of the world and a week later they'll be on diverging paths.
So if the argument is not to judge anyone as a person because everyone would act the same if they had the exact same life circumstances and environment, and everything that affects their decisions fits into life circumstances and environment, what else is left that it would be unfair to judge?
And sure what you said is irrefutable in the sense that it's impossible to collect evidence about it. That's generally a bad sign for theories.