Graphene and borophene are highly attractive two-dimensional materials with outstanding physical properties. In this study we employed combined atomistic continuum multi-scale modeling to explore the effective thermal conductivity of polymer nanocomposites made of polydimethylsilox-ane (PDMS) polymer as the matrix and graphene and borophene as nanofillers. PDMS is a versatile polymer due to its chemical inertia, flexibility and a wide range of properties that can be tuned during synthesis. We first conducted classical Molecular Dynamics (MD) simulations to calculate the thermal conductance at the interfaces between graphene and PDMS and between borophene and PDMS. Acquired results confirm that the interfacial thermal conductance between nanosheets and polymer increases from the single-layer to multilayered nanosheets and finally converges, in the case of graphene, to about 30 MWm−2 K−1 and, for borophene, up to 33 MWm−2 K−1. The data provided by the atomistic simulations were then used in the Finite Element Method (FEM) simulations to evaluate the effective thermal conductivity of polymer nanocomposites at the continuum level. We explored the effects of nanofiller type, volume content, geometry aspect ratio and thickness on the nanocomposite effective thermal conductivity. As a very interesting finding, we found that borophene nanosheets, despite having almost two orders of magnitude lower thermal conductivity than graphene, can yield very close enhancement in the effective thermal conductivity in comparison with graphene, particularly for low volume content and small aspect ratios and thicknesses. We conclude that, for the polymer-based nanocomposites, significant improvement in the thermal conductivity can be reached by improving the bonding between the fillers and polymer, or in other words, by enhancing the thermal conductance at the interface. By taking into account the high electrical conductivity of borophene, our results suggest borophene nanosheets as promising nanofillers to simultaneously enhance the polymers’ thermal and electrical conductivity.