Use a brass drift punch and hammer.

Here's a set of brass drift punches.


$19.95 with free shipping... not too bad a deal.
Brass Drift: Punches | eBay