Once you remove the set screw it does not matter which side you start from!
Gentle taps of a brass punch with the barrel firmly in a padded vise is all it takes.