I always start with a tapered starter punch. You won't move the pin much 'cause of the taper, but that punch won't bend, neither. Finish it off with the smaller one.

I always remove the pin from left to right, right being the side the bolt handle is on.