Next time see if someone with a scope might be willing to spot for you or see if anyone you know can go with you. For me personally, going left to right or up to down on each shot is due to my POA wandering, typically if I shoot a string then reload and shoot another to find them going somewhere else it's because I moved when I settled back down.
The shots you knew you missed the moment you pulled the trigger are easy. The subtle ones are a real pain. Like when you're mid string and drop 4 or 5 shots for no reason then magically pick it back up. What you're talking about bites me the most when shooting the sitting rapid stage... seems like I can never get into the same position I was in when I took my sighters. Coincidentally it's also why I changed how I now test handloads - used to shoot one at a time and check each time on the scope. Now I shoot one for POA/pressure signs, load 5/8/10/whatever and shoot without looking up again.