Starting out as a song plugger in Manhattan's famed Brill Building, Ruby co-wrote hundreds of ephemeral ditties for Broadway shows of the 1910s and 1920s. On the side, Ruby served as comedy writer for Al Jolson and Eddie Cantor.
From the 1930s, Ruby worked in Hollywood. From 1931 to 1935, he headed Warner Bros. Vitaphone short subject department. He also wrote several songs for Hollywood movies, such as "It Must Be Love" for the film "Blond Cheat" (1938), "Only When You're in My Arms" for "The Story of Vernon and Irene Castle" (1939), and "A Poco No" (also known as "Charlie Kane") for "Citizen Kane" (1941). In addition, he contributed to the screenplays of such films as "Words & Music" (1931), "Hot Sands" (1931), "The Old Grey Mayor" (1935), and "Man of Courage" (1943).