The factory uses tape to hold the baffle material in place. I took mine out once just to see what it was like - way too loud for me. As they get older the baffle material "settles in" and they get louder. I really like the lower frequency sound they make. The more baffle you remove, it seems the higher frequency (tinnier) sound goes up proportionately more than the low.
I'm at the point now to where I may add a layer of baffle to take the tinnier sound down a bit. Overall, I think a new set of Specialty Spares sound the best of any I've heard on the Triumph. They're not the loudest out there, but they do allow the engine to breathe and function as it should while sounding great.
You can always put new baffle back in and recreate the original sound at any time. Like Al said, getting the screws back in is definitely the hardest part. Be sure to get 4-stroke baffle (available at most bike shops) if you replace any.