We present a study of the tilt-to-length coupling noise during the LISA Pathfinder mission and how it depended on the system's alignment. Tilt-to-length coupling noise is the unwanted coupling of angular and lateral spacecraft or test mass motion into the primary interferometric displacement readout. It was one of the major noise sources in the LISA Pathfinder mission and is likewise expected to be a primary noise source in LISA. We demonstrate here that a recently derived and published analytical model describes the dependency of the LISA Pathfinder tilt-to-length coupling noise on the alignment of the two freely falling test masses. This was verified with the data taken before and after the realignments performed in March (engineering days) and June 2016, and during a two-day experiment in February 2017 (long cross-talk experiment). The latter was performed with the explicit goal of testing the tilt-to-length coupling noise dependency on the test mass alignment. Using the analytical model, we show that all realignments performed during the mission were only partially successful and explain the reasons why. In addition to the analytical model, we computed another physical tilt-to-length coupling model via a minimizing routine making use of the long cross-talk experiment data. A similar approach could prove useful for the LISA mission.