To me, the hole does look a little oblong like Dave said. I had one that was worn and it did the same thing.

Use a drill bit backward or something similar and push the band spring out from the other side of the stock. Then you can tell if the hole is really oblong or not.

If so, you can fill in the front part to set the spring back some. You might also replace the spring while you're at it. Sometimes the front of the spring wears a little.