Yes, this has been worrying me a bit if I want to try a few different things.And I managed to remove the diode from the lens tube without damage either
Is it best to find a flat ended cylindrical bar that fits snuggly down the front of the tube and press it out squarely on the face of the diode? I suppose you could add a felt/soft face to the bar as well.