The sheet plasmon in epitaxially grown graphene layers on SiC(0001) and the influence of surface roughness have been investigated in detail by means of low-energy electron diffraction (LEED) and electron energy loss spectroscopy (EELS). We show that the existence of steps or grain boundaries in this epitaxial system is a source of strong damping, while the dispersion is rather insensitive to defects. To the first order, the lifetime of the plasmons was found to be proportional to the average terrace length and to the plasmon wavelength. A possible reason for this surprisingly efficient plasmon damping may be the close coincidence of phase (and group) velocities of the plasmons ( almost linear dispersion) with the Fermi velocity of the electrons. Therefore, uncorrelated defects like steps only have to act as a momentum source to effectively couple plasmons to the electron-hole continuum.