There is a huge difference between a Nazi uniform and Germanicon military. Nazi was the political party actually.

German military - army, air force, navy, and then some of the branches wore different colour uniform, eg tank wore black. If somebody questions what you are wearing being Nazi, question their knowledge about WWII uniforms and what uniforms the political party wore. Then ignore them.

Do some research and you will be surprised at what you find.