We discuss various properties of the variational class of continuous matrix product states, a class of Ansatz states for one-dimensional quantum fields that was recently introduced as the direct continuum limit of the highly successful class of matrix product states. We discuss both attributes of the physical states, e.g., by showing in detail how to compute expectation values, as well as properties intrinsic to the representation itself, such as the gauge freedom. We consider general translation noninvariant systems made of several particle species and derive certain regularity properties that need to be satisfied by the variational parameters. We also devote a section to the translation invariant setting in the thermodynamic limit and show how continuous matrix product states possess an intrinsic ultraviolet cutoff. Finally, we introduce a new set of states, which are tangent to the original set of continuous matrix product states. For the case of matrix product states, this construction has recently proven relevant in the development of new algorithms for studying time evolution and elementary excitations of quantum spin chains. We thus lay the foundation for similar developments for one-dimensional quantum fields.