1. Check that the front handguard is not creeping forward under recoil, thus fouling the foresight assembly - this is the prime cause of vertical POI movements in a No4. This is why you often see staked front bands on a No4.
2. Apart from the "down pressure" at the muzzle, ensure that the barrel has free vertical movement inside the barrel channel. Often a "high point" inside one of the handguards is the culprit - the wood coming into contact with the barrel as the rifle heats up.
3. Check the barrel channel itself for high points - sometimes just a lump of old
cosmoline
. One method is to dust the channel and handgaurds with talcum powder - blow off the surplus - then reassemble the rifle with the metal parts lightly oiled. Move the barrel up and down to simulate firing shots, then dissemble the rifle to see if there are any obvious touching points.