Production GPU Training is 34% Slower. Show Me Why

A single slow GPU – a straggler – in a 1,000-node training cluster idles 999 healthy GPUs at every AllReduce barrier. The job does not crash. There is no error message. GPU stragglers just make training run slower than it should – sometimes for hours. This is not hypothetical. Production data from t