We investigate a paradigm example of cavity quantum electrodynamics with many body systems: an ultracold atomic gas inside a pumped optical resonator, confined by the mechanical potential emerging from the cavity-field spatial mode structure. When the optical potential is sufficiently deep, the atomic gas is in the Mott-insulator (MI) state as in open space. Inside the cavity, however, the potential depends on the atomic distribution, which determines the refractive index of the medium, thus altering the intracavity-field amplitude. We derive the effective Bose-Hubbard model describing the physics of the system in one-dimension and study the crossover between the superfluid-MI quantum states. We predict the existence of overlapping stability regions corresponding to competing insulator-like states. Bistable behavior, controlled by the pump intensity, is encountered in the vicinity of the shifted cavity resonance.