The green growth is verdigris, and is the result of the tannic acid in the leather reacting with the copper rivets. If you wipe it off and it get into the leather it is very hard to remove, so the best method of removing it I have found is to remove it with something like a toothpick that will not scratch the leather. Also, the verdigris is poison, so be careful in removing it.
I treat my leather holsters with Pecards, and coating the area where the verdigris is growing with Pecards and letting it set for a few days helps in the removal.