If I remember correctly those colors were part of Army identification at least as far back as the Civil War.
Blue - Infantry
Red - Artillery
And of course the one missing
Yellow - Cavalry