So shave the aluminium. If you do not have a multiple-axis fully programmable milling machine and a tame programmer - why, just try a FILE. Yes, it's slow, but you should only be needing to take off a matter of hundredths of an inch, unless something is seriously bent. The usual criterion for barrel-stock freedom is being able to slide a postcard around everywhere without it binding. I don't see any need for 2mm clearance all round - or have I misunderstood something?
Anyway, I see that you now seem to have made the necessary adjustments. Good luck at the range!