I shoot an original 1851 Cadet smoothbore in N-SSA competition. We are not allowed to use paper wads or cloth patches, just round balls and lube.

There are lots of methods towards decent accuracy with these restrictions. Some competitors coat their bullets with beeswax and/or Alox. Some rasp their projectiles with a farrier's file in order to "fuzz" them up and make them conform to the bore size. Some have custom molds made.

My musket has a bore like a drain pipe. It is .569 at it's narrowest point. I use a coated .562 ball over 42 grains of 3F powder. Off the bench @ 25 yards, it will shoot a ragged but one holed group. At 50 yards off hand it will shoot within a 6 x6 inch square until the heat mirage coming off the barrel makes it impossible to see the target. All this from a gun with no rear sight - and I'm not particularly competitive anymore.

The point I'm trying to make is that the whole "wobbling down the barrel and going who knows where" issue is, if not quite an urban myth, then at least easily avoidable (at least out to 75 yards or so