The main purpose of this paper is to present a conceptual approach to understanding the extension of the Prym map from the space of admissible double covers of stable curves to different toroidal compactifications of the moduli space of principally polarized abelian varieties. By separating the combinatorial problems from the geometric aspects we can reduce this to the computation of certain monodromy cones. In this way we not only shed new light on the extension results of Alexeev, Birkenhake, Hulek, and Vologodsky for the second Voronoi toroidal compactification, but we also apply this to other toroidal compactifications, in particular the perfect cone compactification, for which we obtain a combinatorial characterization of the indeterminacy locus, as well as a geometric description up to codimension six, and an explicit toroidal resolution of the Prym map up to codimension four. © 2017 European Mathematical Society.