Marine infrastructures such as harbours, bridges, and locks are particularly exposed to salt water and are therefore subject to increasing deterioration. This makes regular inspection of the structures necessary. The inspection is carried out manually, using divers under water. To improve this costly and time-consuming process, we propose to scan the surface and underwater structure of the port with a multi-sensor system (MSS) and classify the obtained point cloud into damaged and undamaged areas fully automatically. The MSS consists of a high-resolution hydro-acoustic underwater multi-beam echo-sounder, an above-water profile laser scanner, and five HDR cameras. In addition to the IMU-GPS/GNSS method known from various applications, hybrid referencing with automatically tracking total stations is used for positioning. The key research idea relies on 3D data from TLS, multi-beam or dense image matching. For this purpose, we build a rasterised heightfield of the point cloud of a harbour structure by reducing the CAD-based geometry from the measured 3D point cloud. To do this, we fit regular shapes into the point cloud and determine the distance of the points to the geometry. To detect anomalies in the data, we use two methods in our approach. First, we use the VGG19 Deep Neural Network (DNN), and second, we use the Local-Outlier-Factors (LOF) method. To test and validate the developed methods, training data was simulated. Afterwards, the developed methods were evaluated on real data set in Lübeck, Germany, which were acquired with the developed Multi-Sensor-System (MSS). In contrast to the traditional, manual method by divers, we have presented an approach that allows for automated, consistent, and complete damage detection. We have achieved an accuracy of 90.5% for the method. The approach can also be applied to other infrastructures such as tunnels and bridges.