Haynes began his full time professional career in 1945. From 1947 to 1949 he worked with saxophonist Lester Young, and from 1949 to 1952 was a member of saxophonist Charlie Parker's quintet. He also recorded at the time with pianist Bud Powell and saxophonists Wardell Gray, and Stan Getz. From 1953 to 1958 he toured with singer Sarah Vaughan. Haynes went on to work with more experimental musicians, like saxophonists John Coltrane and Eric Dolphy, or pianists Chick Corea and Andrew Hill.
Haynes has recorded or performed with Gary Burton, Miles Davis, Dizzy Gillespie, Henry Grimes, Christian McBride, Jackie McLean, Pat Metheny, Thelonious Monk, Gerry Mulligan, Art Pepper, Horace Tapscott and many others. He has also led his own groups, some performing under the name Hip Ensemble. His most recent recordings as a leader are Fountain of Youth and Whereas, both of which have been nominated for a Grammy Award. Well into late 2008, Haynes was still continuing to perform worldwide and on December 22, 2010, he was named a recipient of a Grammy Lifetime Achievement Award by the National Academy of Recording Arts and Sciences. Haynes received the award at the Special Merit Awards Ceremony & Nominees Reception of the 53rd Annual Grammy Awards on February 12, 2011.
His son Craig Haynes is a drummer with the Sun Ra Arkestra, his son Graham Haynes is a cornetist, and his grandson Marcus Gilmore and nephew Christopher Haynes are both drummers.