Tiny measurements that make a preload longer or one fork a tad shorter are unlikely to be noticed unless you are racing...I'd guess that maybe the rear wheel might be a skosh out of square before I'd buy into the forks causing it to pull right. Sounds a whole lot more like a cupped tire wobble combined with a tiny bit of alignment distress.
So how much sag did you end up with using the new spacers? By lowering the fork the suspension has a little less travel (by virtue of the fender hitting the triple tree) so you might need to recalculate. About 30% of the total travel or 25% if you ride more racer style (to make the forks quicker in the turns, the fork dive changes the angle faster).
So have you loosened the axle, bounced the front end and retorqued? Always a possibility of a minor alignment prob there when loosening the tubes in the tree.
