This paper is the second in a set of two investigating tilt-to-length (TTL) coupling. TTL describes the cross-coupling of angular or translational jitter into an interferometric phase signal and is an important noise source in precision interferometers, including space gravitational wave detectors like LISA. We discussed in Hartig et al (2022 J. Opt. 24 065601) the TTL coupling effects originating from optical path length changes, i.e. geometric TTL coupling. Within this work, we focus on the wavefront and detector geometry dependent TTL coupling, called non-geometric TTL coupling, in the case of two interfering fundamental Gaussian beams. We characterise the coupling originating from the properties of the interfering beams, i.e. their absolute and relative angle at the detector, their relative offset and the individual beam parameters. Furthermore, we discuss the dependency of the TTL coupling on the geometry of the detecting photodiode. Wherever possible, we provide analytical expressions for the expected TTL coupling effects. We investigate the non-geometric coupling effects originating from beam walk due to the angular or translational jitter of a mirror or a receiving system. These effects are directly compared with the corresponding detected optical path length changes in Hartig et al (2022 J. Opt. 24 065601). Both together provide the total interferometric readout. We discuss in which cases the geometric and non-geometric TTL effects cancel one-another. Additionally, we list linear TTL contributions that can be used to counteract other TTL effects. Altogether, our results provide key knowledge to minimise the total TTL coupling noise in experiments by design or realignment.