I wouldn't touch the shoulder at all. Lap the washer with fine emery paper on a dead flat surface such as a piece of heavy plate glass, lapping plate or gauge block. A piece of very fine-grained granite countertop can be a cheap option too; a sink cut-out for example.
That way you are taking off thousandths at a time and can't go too far, too quickly as can easily happen in a lathe, apart from the alignment issues with a round, tapered exterior.