Berger studied with Franz Schmidt at the Wiener Musikakademie (now the University of Music and Performing Arts, Vienna) from 1926 to 1932. From 1932 to 1939 he was in Berlin, where Wilhelm Furtwängler became an active proponent of his work. He returned to Vienna in 1939 and remained there for the rest of his life, with extended stays in Germany and the United States. In later years, although he continued to compose and publish new works, his music was less frequently performed, and he largely withdrew from public musical life.
Most of Berger's published music is orchestral. His distinct musical language is characterized by shifting tonalities, complex rhythms, inventive and nuanced orchestration, and innovative structural principles.