Slug your barrel. Not all No. 4 Rifles have a .311" barrel. They can go to .315" and be considered ok.
The issue is the American manufacturers don't use the same diameter. Hornady uses .312" diameter or .3105". Sierra uses .311"(they make a .311" MatchKing). You'll have to measure a Remington bullet as they don't publish what diameter they use. Prvi uses .311". However, none of that matters if you have a .313" + barrel.
Montana Bullets makes several over .313" cast bullets that aren't stupid expensive.
In any case, if you're not reloading you have to try a box of as many brands as you can to find the ammo that rifle likes. But you must slug the barrel on any Lee-Enfield first.