Mills was born to Jewish parents in the Lower East Side of Manhattan in New York City. He founded Mills Music with his brother Jack in 1919. Between 1919 and 1965, when they sold Mills Music, Inc., they built and became the largest independent music publisher in the world.
As a music scout, Mills signed Duke Ellington and Cab Calloway. Mills is highly important in the history of jazz music because of his willingness to work with black musicians and publish their music. While Mills is mostly remembered for his business abilities, he more than dabbled in songwriting -- included among his works is the hit It Don't Mean a Thing if It Ain't Got That Swing.
He died in 1985 in Palm Springs, California and is buried at Mount Sinai Memorial Park, Los Angeles, CA.