If anything would be damaged upon removal it would be tearing the insert because it's glued to the bar. Kuryakyn offers replacement rubber grip inserts for their ISO grips, so their is nothing to worry about. I haven't tried removing mine, so I can't say if it's an issue or not.