Definitely go spandex, or even a pair of slippery mesh shorts over the boxers/under the pants helps. The problem is the twisting of your skin. Having a slippery layer in there allows your pants to conform to your seat, and your skin/bum to settle into its own shape. Make sense?
