He was married to
Michèle Morgan from 1942 to 1948 (divorced)
Micheline Presle from 1950 to 1954 (divorced), their daughter is Tonie Marshall and to
Ginger Rogers from 1961–1969 (divorced).
He became a vocalist for Fred Waring and his band, the "Pennsylvanians " before forming his own band in 1937.