Interesting, I did not think of the turbulence the exhaust was creating could produce a pull.
Not the exhaust turbulance but the added frontal area and surface area in the air on the side with the pipe hanging out adds drag on that side.
Check your rear swingarm and axle for play and how well you rear wheel lines up overall, just to be sure. At lower speeds you are also in a lower gear and can put more force down.