Born in Cinco Hollow in Hocking County, Ohio, he was taken as a child to Jellico in eastern Tennessee and there worked with his father in the lumber mill business. Although he learned the mountain ballads, he preferred Negro spirituals because they emphasized harmony and rhythm and had a "definite religious purpose." Rodeheaver early learned to play the cornet but switched to trombone while attending Ohio Wesleyan College, where he also served as a cheerleader.
In 1898 he left college to serve in the Fourth Tennessee Band during the Spanish-American War. Around 1904 he joined evangelist William E. Biederwolf as music director and then served, from 1910 to 1930, in the same role for Billy Sunday, the most popular evangelist of the period. (Shortly after Billy Sunday's death in 1935, Rodeheaver wrote a memoir of his relationship with the evangelist.)
Music Director for Billy Sunday
When Lowell Thomas presented Rodeheaver to the New York Advertising Club, Rodeheaver succeeded in getting the advertising agents to sing "Pray the Clouds Away." Will Rogers said, "Rody is the fellow that can make you sing whether you want to or not. I think he has more terrible voices in what was supposed to be unison than any man in the world. Everyone sings for Rody!" When Rodeheaver was introduced to John D. Rockefeller, Sr., on a golf course, Rockefeller delayed his golf game long enough to sing with Rodeheaver, "I'll Go Where You Want Me to Go, Dear Lord." In 1940, Rodeheaver led the singing for 250,000 people who attended the Wendell Willkie homecoming in Elwood, Indiana.
In the days before electronic amplification, Rodeheaver quickly discovered that his trombone could be heard when his voice or the piano could not. He often led congregational singing with his trombone, switching from playing to directing half way through the song and then allowing the trombone to hang on his arm at the elbow. During a Sunday tent campaign in Kansas, a heavy storm with near-hurricane winds caused the top and sides to sag, and a quarter pole fell, striking a woman on the head. When the crowd panicked and rose to flee, Rodeheaver began playing his trombone and the crowd quieted.
In his prime, Rodeheaver also used his baritone voice to good effect as a soloist and as a participant in ensembles composed of other members of Sunday’s evangelistic team—especially duets with contralto Virginia Asher. During the heyday of the Sunday evangelistic campaigns, Rodeheaver directed the nation’s largest choruses: from a few hundred to as many as two thousand volunteers in Sunday’s various campaigns. To him there was nothing incongruous about having his choirs sing the gospel song "Master, the Tempest is Raging", followed by the Hallelujah Chorus from Handel's Messiah.
In 1913 Rodeheaver began recording for the Victor Talking Machine Company, a relationship that lasted for twenty years. He also recorded for Gennett, Columbia and for his own Rainbow Records label. Some of his records, such as "The Unclouded Day" and "The Great Judgement Morning," were so popular that they had to be rerecorded to keep up with demand. Other records featured Rodeheaver's recitations of sentimental poetry, such as Paul Laurence Dunbar's "When Malindy Sings" (1916).
Rodeheaver appeared on at least eighteen record labels and five hundred sides during his recording career. His most recorded piece was Sunday’s theme song "Brighten The Corner Where You Are," a bouncy number of extremely limited theological depth. Rodeheaver recorded it for at least 17 different labels. Rodeheaver’s other most recorded titles were "Mother's Prayers Have Followed Me", "If Your Heart Keeps Right", "The Old Rugged Cross", "Since Jesus Came Into My Heart", "In The Garden", and "My Wonderful Dream".
In 1910, Rodeheaver started his own publishing business, the Rodeheaver Company, compiling gospel songs to sell at revivals. In 1936 Rodeheaver purchased the Hall-Mack Company and merged it with his own publishing house, headquartered in Winona Lake, Indiana. Rodeheaver employed songwriters such as B. D. Ackley and Charles H. Gabriel to write songs for his company, but he also composed a number of tunes himself, including most notably, “When Jesus Came.” Around 1922, his company began issuing 78-rpm records on its own Rainbow label, the nation's first record company devoted solely to gospel music.
Billy Sunday perhaps paid Rodeheaver $80,000–90,000 over the course of their twenty-year partnership, but Rodeheaver admitted that he made more than four times that amount from other sources, especially music publishing, during those same years