The trick to smooth bore accuracy is velocity - you need to be using towards the heaviest charge your musket will take. Muskets are much less efficient than rifles and consequently need larger charges.. typical Brown Bess service loads were in the 100gn + range.

Also the bullets need to be as soft as you can make them.. 100% pure lead - no wheelweights or solder!