Always helps to have a bigger hammer at hand! I have one of those semi mallets that is plastic on one side and hard rubber on the other for such jobs.