I don't know if there were any factory prototypes designed for alternate cartridges, but certainly the commercial community was active during the 1960's with a great many conversions done to the standard action in many commercial and some wildcat cartridges.
Within the limitations of the M1 Carbines action length, there’s not much that can be done to improve performance with alternate cartridges, unless you’re looking at a flatter shooting round ie. The 5.7 Johnson, 22/30 Carbine, or perhaps the best of the bunch, the .22 Gustafson (basically a shortened .221 Fireball round).
You could go the other way, which would entail chambering for one of the big bore variants. Many were tried by various gunsmiths in the 60’s up to and including .45 colt. Probably the most successful of the bunch commercially, (and it wasn’t really all that successful as only about 1500 were made) was the Universal Vulcan 440. This was a spring assisted Pump Action .44 Magnum M1 Carbine derivative manufactured by Universal Firearms in Hialeah Florida from about 1964-67. You can still find these for sale on the usual auction sites, but good luck finding parts!
I imagine you’re thinking that pretty much everything has been tried on the Carbine action. I thought so too, until I came across a posting on “Shooters forum” about a fellow who calls himself “Grumpy OlGuy” who did something rather novel.
Here it is in his own words:
OCD caused me to try this: 110gr Spitzer BC=.273. With ~1970fps/960 ft-lbf at the muzzle, gives 9mm parabellum muzzle velocity and energy at ~350+yds (~1200 fps, ~350 ft-lbf; about 100+ yards greater range performance than the standard carbine ballistics). Drops to subsonic at about 400 yds. That's a heck of a pistol round. (I don't think there's a way to post a photo)
Uses 223 parent case, shortened 300 BO dies (to form case), and a 300 BO chamber reamer with a standard barrel. Case is shortened to 1.185in. OAL cartridge is 1.685, so can load into a standard magazine (but only 10 or 11 rounds). Case volume is about the same as the standard, although the bullet seats a little deeper (the bullet is about .125 longer than the standard FMJ). Test results so far (around 50 rnds) -- cycles ok with 12.0 to 13.0 gr WC820; ~2-2.5 MoA. Chronoed at ~1920fps. Haven't tried any higher loadings. My normal carbine load is ~13.2-13.4gr WC820 with a 110gr FMJ. Used a barrel from Sarco. Next up is to try some lil' gun. I'd love to get this up over 1000 ft-lbf and see if the accuracy was ok.
The point was to see if one could extend the effective range of the carbine and stay within the overall cartridge length. And to show what might have been possible in 1940. (Not that the Carbine didn't fulfill its intended objective extremely well. Just could have been a little more with a different bullet). For those of us who really appreciate the way this thing handles. I think 9mm ballistic/power presents a reasonable motivation to keep one's head down.