We analyze the impact of a proposed tidal instability coupling p modes and g modes within neutron stars on GW170817. This nonresonant instability transfers energy from the orbit of the binary to internal modes of the stars, accelerating the gravitational-wave driven inspiral. We model the impact of this instability on the phasing of the gravitational wave signal using three parameters per star: An overall amplitude, a saturation frequency, and a spectral index. Incorporating these additional parameters, we compute the Bayes factor (lnB!pgpg) comparing our p-g model to a standard one. We find that the observed signal is consistent with waveform models that neglect p-g effects, with lnB!pgpg=0.03-0.58+0.70 (maximum a posteriori and 90% credible region). By injecting simulated signals that do not include p-g effects and recovering them with the p-g model, we show that there is a ≃50% probability of obtaining similar lnB!pgpg even when p-g effects are absent. We find that the p-g amplitude for 1.4 MâŠneutron stars is constrained to less than a few tenths of the theoretical maximum, with maxima a posteriori near one-Tenth this maximum and p-g saturation frequency ∼70 Hz. This suggests that there are less than a few hundred excited modes, assuming they all saturate by wave breaking. For comparison, theoretical upper bounds suggest a103 modes saturate by wave breaking. Thus, the measured constraints only rule out extreme values of the p-g parameters. They also imply that the instability dissipates a1051 erg over the entire inspiral, i.e., less than a few percent of the energy radiated as gravitational waves. © 2019 American Physical Society.