Starfleetmarines rarely appeared in any of the incarnations of Star Trek, although Gene Roddenberry once suggested that, in the original Star Trek series, the Enterprise carried a platoon of Starfleet Marines. However, he never got the chance to show them on screen in an episode.
The first onscreen appearance of Starfleet Marines was in Star Trek V: The Final Frontier (the Starfleet officers accompanying Kirk and company to Nimbus III have since been said to be Marines) and Star Trek VI: The Undiscovered Country, with the appearance of Colonel West. Starfleet marines also appeared in many episodes of Star Trek: Deep Space Nine (they wear one-piece black uniforms).
In Star Trek: Enterprise, MACOs - Military Assault Command Operations appeared, however they weren't part of Starfleet. It is likely that, with the formation of the United Federation of Planets (and the founding of Starfleet Academy), the MACO teams will be absorbed into Starfleet and become the Starfleet marines.