As frequency bands become increasingly crowded, one of the major research challenges for future wireless communication networks is how to use the radio frequency bands more efficiently. In wireless communication networks, spectral efficiency can be improved either through frequency reuse technologies such as cognitive radio and multi-user techniques or by using technologies to increase the data rate. However, in this regard there is still room for further improvements, and thus, development of novel hybrid solution approaches to enhance spectral efficiency for future wireless communication networks is put into the focus of this dissertation.In this thesis, a multi-tier heterogeneous network with macro cells, small cellsand relays is addressed. In order to improve spectral efficiency in a single carriersystem, the non-orthogonal transmission scheme Faster-than Nyquist (FTN) signaling is investigated. Furthermore, novel hybrid interference mitigation approaches to achieve a spectral efficiency improvement in multi-user multicarrier Multiple-Input and Multiple-Output (MIMO) systems by using cognitive radio and relay technology are proposed.