He became famous for his unusually high contralto voice, which was due to Kallmann's syndrome, a very rare genetic condition. The condition stunted his growth at four feet eleven inches until, at the age of 37, he grew another eight inches to the height of five feet seven inches. The condition prevented him from reaching puberty, leaving him with a high, undeveloped voice.
The singer's big break came in 1949, when Lionel Hampton hired him on the recommendation of Paul Gayten and billed him as "Little Jimmy Scott" and had a hit with "Everybody's Somebody's Fool". This song and three others recorded with the Hampton orchestra, along with early 1950's solo sides for the Coral and Brunswick labels, were reissued in 1999 on the GRP CD "Everybody's Somebody's Fool". Also released that year was the three-CD "The Savoy Years and More" containing his 1952 recordings for Roost Records and his 1955-75 output for Savoy Records.