We have studied the single-electron transport spectrum of a quantum dot in GaAs/AlGaAs resonant tunneling device. The measured spectrum has irregularities indicating a broken circular symmetry. We model the system with an external potential consisting of a parabolic confinement and a negatively charged Coulombic impurity placed in the vicinity of the quantum dot. The model leads to good agreement between the calculated single-electron eigenenergies and the experimental spectrum. Furthermore, we use the spin-density-functional theory to study the energies and angular momenta when the system contains many interacting electrons. In the high magnetic field regime the increasing electron number is shown to reduce the distortion induced by the impurity.