The main issue is not point of impact, it's gas port pressure. Current .30-06 factory loadings generally use a powder with a slower burn rate and that elevates gas port pressure to a point that it is possible to damage the rifle (M1that is).
Granted a few rounds may not do much damage but if the receiver is not anealed or if some other components are "iffy" even one "over-pressure" round may wreck the rifle.