Phase plays a crucial role in many quantum effects including interference. Here we lay the foundations for the study of phase in probabilistic theories more generally. Phase is normally defined in terms of complex numbers that appear when representing quantum states as complex vectors. Here we give an operational definition whereby phase is instead defined in terms of measurement statistics. Our definition is phrased in terms of the operational framework known as generalized probabilistic theories or the convex framework. The definition makes it possible to ask whether other theories in this framework can also have phase. We apply our definition to investigate phase and interference in several example theories: classical probability theory, a version of Spekkens' toy model, quantum theory and box-world. We find that phase is ubiquitous; any non-classical theory can be said to have non-trivial phase dynamics.