The topographic amplification effect has significant impacts on structural safety since it causes inconsistencies in seismic response. Previous studies on topographical influence focus on a certain point and cross-sections, which is insufficient to comprehensively understand the topographic amplification effect. Besides, few studies analyze the relationship between seismic response and ground motion frequency. Hence, this study aims to investigate the effects of topographic irregularity of the whole site, and explore the relationship between seismic response characteristics and signal frequency. An analysis procedure for modeling and meshing an actual 3D site is proposed by combining SolidWorks, SketchUp and Abaqus. Finite element method (FEM) is applied to simulate the seismic response. Results show that the site amplification and de-amplification are associated with topographic features. Specifically, the seismic responses of ravine and ridge areas could differ by a factor of two. However, acceleration response in ravine areas shows unusual amplification under input signal frequencies over 4 Hz. Besides, the frequency of input ground motion certainly affects the seismic site amplification. Specifically, in this study, the seismic response peaks when the input frequency is in the range of 0.4–0.8 Hz, which corresponds to the resonance frequency of the numerical model. In terms of frequency, site amplification showed a correlation between actual recordings and artificial signal inputs, but lower intensities are observed when actual records are input. Therefore, the engineering risk will be underestimated when the topographic irregularity and frequency characteristic of input ground motion are inappropriately considered. The findings of this study provide a new approach to investigate the actual 3D site amplification effects and shed new insight on regional seismic risk analysis.