I wouldn't worry about it till you measure to see if you even need any shims. Then worry The only PITA you'll save by not removing the cams is the potential for breaking the bolts that hold the cams in.
Benny
Black & Silver '02
Too many mods to list
Not enough miles ridden