Optomechanical systems can exhibit self-sustained limit cycles where the quantum state of the mechanical resonator possesses nonclassical characteristics such as a strongly negative Wigner density, as was shown recently in a numerical study by Qian et al. [Phys. Rev. Lett. 109, 253601 (2012)]. Here, we derive a Fokker-Planck equation describing mechanical limit cycles in the quantum regime that correctly reproduces the numerically observed nonclassical features. The derivation starts from the standard optomechanical master equation and is based on techniques borrowed from the laser theory due to Haake and Lewenstein. We compare our analytical model with numerical solutions of the master equation based on Monte Carlo simulations and find very good agreement over a wide and so far unexplored regime of system parameters. As one main conclusion, we predict negative Wigner functions to be observable even for surprisingly classical parameters, i.e., outside the single-photon strong-coupling regime, for strong cavity drive and rather large limit-cycle amplitudes. The approach taken here provides a natural starting point for further studies of quantum effects in optomechanics.