The band is composed of several very recognizable faces in the new-school punk rock scene, including Spike Slawson of the $wingin' Utter$ , Fat Mike from NOFX, Chris Shiflett of Foo Fighters (and formerly No Use For A Name), as well as Joey Cape and Dave Raun of Lagwagon. The band wasn't originally planning on making an album, appearing only on compilation albums for their first year. In 1996, however, the band began releasing singles named after the singer they had covered. They produced the 7" singles "Billy" (Joel), (Neil) "Diamond", "Paul" (Simon), (John) "Denver", and "Barry" (Manilow) in this manner. Their first album, “Have a Ball”, was released in 1997. The band still performs whenever its members aren't busy with their other bands, and makes plenty of guest appearances on punk compilations.
Each album by the band is themed differently: “Have a Ball” focuses on classic 1970s and early 1980s songs by artists like Elton John, Neil Diamond, and John Denver; “Are a Drag” consists entirely of showtunes,Blow In The Wind consists of 60's classics, Take A Break consists entirely of R&B songs by artists such as Boyz II Men, Lionel Richie, and Vanessa Williams; their fifth album, “Ruin Jonny's Bar Mitzvah”, was recorded live at an actual bar mitzvah and consists of pop music from the 1960s through the 1980s by artists such as REO Speedwagon, Styx and The Beatles, as well as traditionals like "Hava Nagila".
Me First and The Gimme Gimmes are also known for filling their covers with famous guitar riffs from songs by other punk rock bands. For instance, Elenor begins with the opening tune of The Clash's London Calling, My Favourite Things begins with the melody and opening lyrics of Bad Religion's “Generator” and on the song Hava Nagila (Christmas Arrangement), the melody from The Offspring ’s “Come Out And Play” can be heard.
They entered the studio on April 3, 2006 to work on their sixth album, “Love Their Country”, which was released on October 17, 2006. The theme of this album is country and western, and includes covers of tracks by Dixie Chicks, Garth Brooks, Hank Williams and Johnny Cash. Prior to the release of the album, Fat Wreck Chords released a digital label sampler iFloyd which included "(Ghost) Riders in the Sky" by the band.