As Peter says, the normal movement in sight components will not make any difference at the target.
If you have good groups, but they move around the target, then the rifle has some sort of bedding problem. I would guess that maybe:
1. The front handguard is moving - and obstructing the movement of the barrel by touching the foresight protector;
2. There is something in the barrel channel which is causing the barrel to stick. Sometimes "moving groups" are the result of a lump of cosmolinein the barrel channel: when the rifle is cold it shoots in one place, but then as the barrel gets hot, the cosmoline melts - and the barrel moves its point of impact somewhere else.