Current dual-frequency magneto-optical traps (MOTs) for ultracold molecules are plagued by sub-Doppler heating effects, making them vastly inferior to standard atomic MOTs. Here we demonstrate theoretically that the sub-Doppler effects in such a MOT can be engineered to provide cooling instead of heating. We give an intuitive picture how to achieve such cooling and show the cooling and trapping force results of the 16-level optical Bloch equations for the case of CaF molecules. From three-dimensional Monte Carlo simulations we estimate the temperature and density of our MOT to be 40μK and 4×108cm-3, respectively, for a molecule number of 1×105. We also extend our idea to the case of atomic MOTs and show that it can be used to produce sub-Doppler forces in these systems that are much more robust against magnetic fields.