Brown was born in Cordele, Georgia. In the 1930s and 1940s he played harmonica at local clubs and made some non-commercial recordings, such as "War Song" and "I'm Gonna Make You Happy" (1943), recorded for the Library of Congress' Folk Music Archive.
Brown moved to New York in 1956, where he was discovered by Fire Records (2) owner Bobby Robinson. In 1959 Brown recorded the rustic blues, "Fannie Mae", which went to # 38 in the U.S. Top 40, and to #1 on the R&B chart in April 1960. In later years he recorded for Checker Records and for numerous small record labels.
One of his song, "Doctor Brown" was covered by Fleetwood Mac (Mr. Wonderful).
His song "Fannie Mae" enjoyed further attention in 1973 when it was included in the film American Graffiti and the best selling soundtrack 41 Original Hits From The Sound Track Of American Graffiti.
The songwriting credits for his songs, "Fannie Mae" or "Doctor Brown", may vary from one edition to another, with the credits sometime going to Waymon Glasco. It is claimed Waymon Glasco was his manager who bought his publishing rights after his death while other sources claim that was Buster Brown's real name.