I am a bit confused about how you are using the rest. Usually with a sporting type rifle I put my left hand underthe forend and rest the back of my hand on the rest. I aim the rifle carefully for every shot, holding the rifle quite firmly.

Your description makes it sound like you are not aiming after the first shot - but I must be misreading that part.