I feel your pain. I do not know how they put these guys on but the one I had to remove I had to cut off it was on that tight. In an ideal world it should come right off when the cross pin is removed but mine did not. The key sits in a groove so trying to remove it will not work. I think the big problem is tapping the front sight puts unequal pressure on the sight and causes a jam. If you can try tapping the lower edge of the sight with a brass punch in addition to tapping the upper part.

Good luck!!