Simon's professional career began when he moved to Oakland, California and released, "My Adorable One" in 1962, which was a minor hit. In 1964, Joe met John Richbourg, a Nashville-based disc jockey who began guiding the singer's musical path, initially on the Sound Stage 7 label. "Let's Do It Over" (1965), Simon's first R&B hit, emphasized Richbourg's preference for a blend of gentle soul and country, and the singer's smooth delivery found its niche on such poignant songs as "Teenager's Prayer", "Nine Pound Steel" (both 1967) and "The Chokin' Kind", a US R&B number 1 in 1969.
The following year, Simon moved to the Polydor subsidiary Spring Records. He maintained his ties with Richbourg until 1971, when a Gamble & Huff production, "Drowning In The Sea Of Love", was an R&B number 3. Further success came with "The Power Of Love" (1972), "Step By Step" (1973 - his only UK hit), "Theme From Cleopatra Jones" (1973) and "Get Down Get Down (Get On The Floor)" (1975), but the artist increasingly sacrificed his craft in favour of the dance floor. His late 1970s releases were less well received and in 1980 he returned to Nashville. From that time, Simon's work was restricted to local labels.
At the beginning of the 1980s, Simon joined the Posse Records label, making a return to country soul. He signed with Compleat Records in the mid-1980s. Simon’s last secular 45 found him back with Richbourg cutting the Tony Joe White penned "Alone At Last"/ “Stay Love”. When Richbourg passed away Simon lost heart and turned away from the secular music world and devoted his life to Christianity becoming an evangelist preacher.