Ultralight bosons can form large clouds around stellar-mass black holes via the superradiance instability. Through processes such as annihilation, these bosons can source continuous gravitational-wave signals with frequencies within the range of LIGO and Virgo. If boson annihilation occurs, then the Galactic black hole population will give rise to many gravitational signals; we refer to this as the ensemble signal. We characterize the ensemble signal as observed by the gravitational-wave detectors; this is important because the ensemble signal carries the primary signature that a continuous wave signal has a boson annihilation origin. We explore how a broad set of black hole population parameters affects the resulting spin-0 boson annihilation signal and consider its detectability by recent searches for continuous gravitational waves. A population of 108 black holes with masses up to 30M and a flat dimensionless initial spin distribution between zero and unity produces up to 1000 signals loud enough in principle to be detected by these searches. For a more moderately spinning population, the number of signals drops by about an order of magnitude, still yielding up to 100 detectable signals for some boson masses. A nondetection of annihilation signals at frequencies between 100 and 1200 Hz disfavors the existence of scalar bosons with rest energies between 2×10-13 and 2.5×10-12 eV. Finally, we show that, depending on the black hole population parameters, care must be taken in assuming that the continuous wave upper limits from searches for isolated signals are still valid for signals that are part of a dense ensemble: Between 200 and 300 Hz, we urge caution when interpreting a null result for bosons between 4×10-13 and 6×10-13 eV. © 2020 authors. Published by the American Physical Society. Published by the American Physical Society.