We investigate both experimentally and theoretically disorder-induced damping of Bloch oscillations of Bose-Einstein condensates in optical lattices. The spatially inhomogeneous force responsible for the damping is realized by a combination of a disordered optical and a magnetic gradient potential. We show that the inhomogeneity of this force results in a broadening of the quasimomentum spectrum, which in turn causes damping of the centre-of-mass oscillation. We quantitatively compare the obtained damping rates to the simulations using the Gross-Pitaevskii equation. Our results are relevant for high precision experiments on very small forces, which require the observation of a large number of oscillation cycles.