Adsorption-induced reduction of surface-state conductivity in epitaxial Bi(111) films, a prototype system with large Rashba-induced surface-state splitting, by adsorbed atoms of Bi, Fe, and Co has been investigated by macroscopic surface magnetotransport measurements at a temperature of 10 K. A detailed analysis of magnetotransport, dc transport, and Hall data reveals that the scattering efficiencies for Co and Fe are larger by a factor of 2 than that for Bi. While for the latter charge transfer and change of band filling near the Fermi level are negligible, we find an increase of hole concentration upon Co and Fe adsorption. These atoms act as acceptors and immobilize on average about 0.5 electrons per adsorbed atom. Besides the dominant classical magnetoconductance signal the films show signatures of weak antilocalization, reflecting the strong spin-orbit coupling in Bi(111) surface states. This behavior can be changed to weak localization by the adsorption of high concentrations (0.1 monolayers) of magnetic impurities (Fe,Co), similarly to results found on the topological insulator Bi2Se3. Our results demonstrate that details of chemical bond formation for impurities are crucial for local spin moments and electronic scattering properties. © 2012 American Physical Society.