Contact interaction of two bodies can be modeled using the penalty function approach while its accuracy and robustness are directly associated with the geometry of contact bodies. Particularly, in the research fields of rock mechanics, we need to treat polygonal shapes such as mineral grains/particles at a mesoscale and rock blocks at a macroscale. The irregular shapes (e.g., polygons with small angles or small edges) pose challenges to traditional contact solution approach in terms of algorithmic robustness and complexity. This paper proposed a robust potential-based penalty function approach to solve contact of polygonal particles/block. An improved potential function is proposed considering irregular polygonal shapes. A contact detection procedure based on the entrance block concept is presented, followed by a numerical integral algorithm to compute the contact force. The proposed contact detection approach is implemented into discontinuous deformation analysis with an explicit formulation. The accuracy and robustness of the proposed contact detection approach are verified by benchmarking examples. The potential of the proposed approach in analysis of kinetic behavior of complex polygonal block systems is shown by two application examples. It can be applied in any discontinuous computation models using stepwise contact force-based solution procedures. © 2020, The Author(s).