Pollination services of cacao are crucial for global chocolate production, yet remain critically understudied, particularly in regions of origin of the species. Notably, uncertainties remain concerning the identity of cacao pollinators, the influence of landscape (forest distance) and management (shade cover) on flower visitation and the role of pollen deposition in limiting fruit set. Here, we aimed to improve understanding of cacao pollination by studying limiting factors of fruit set in Peru, part of the centre of origin of cacao. Flower visitors were sampled with sticky insect glue in 20 cacao agroforests in two biogeographically distinct regions of Peru, across gradients of shade cover and forest distance. Further, we assessed pollen quantities and compared fruit set between naturally and manually pollinated flowers. The most abundant flower visitors were aphids, ants and thrips in the north and thrips, midges and parasitoid wasps in the south of Peru. We present some evidence of increasing visitation rates from medium to high shade (40%–95% canopy closure) in the dry north, and opposite patterns in the semi-humid south, during the wet season. Natural pollination resulted in remarkably low fruit set rates (2%), and very low pollen deposition. After hand pollination, fruit set more than tripled (7%), but was still low. The diversity and high relative abundances of herbivore flower visitors limit our ability to draw conclusions on the functional role of different flower visitors. The remarkably low fruit set of naturally and even hand pollinated flowers indicates that other unaddressed factors limit cacao fruit production. Such factors could be, amongst others, a lack of effective pollinators, genetic incompatibility or resource limitation. Revealing efficient pollinator species and other causes of low fruit set rates is therefore key to establish location-specific management strategies and develop high yielding native cacao agroforestry systems in regions of origin of cacao. © 2022 The Authors. Ecological Solutions and Evidence published by John Wiley & Sons Ltd on behalf of British Ecological Society.