Well, but that's exactly what A/B testing should be used for - instead of optimizing to a local optimum, you use it to verify if the 'new and improved' checkout or search or navigation is really an improvement, or is insignificant, or happens to make things worse.

"The obvious things I know will have an impact" sometimes don't have an impact and sometimes have a negative impact. Experience makes the chances better, but nowhere near perfect.